You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Varun Sharma (JIRA)" <ji...@apache.org> on 2012/11/03 01:24:11 UTC
[jira] [Created] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Varun Sharma created HBASE-7093:
-----------------------------------
Summary: Couple Increments/Appends with Put/Delete(s)
Key: HBASE-7093
URL: https://issues.apache.org/jira/browse/HBASE-7093
Project: HBase
Issue Type: Improvement
Affects Versions: 0.96.0
Reporter: Varun Sharma
See related issue - https://issues.apache.org/jira/browse/HBASE-4583
Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
One possible high level approach could be:
1) Class IncrementMutation which inherits from Increment and Mutation
2) In the mutateRow call, we add a case for "IncrementMutation" object
3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498205#comment-13498205 ]
Andrew Purtell commented on HBASE-7093:
---------------------------------------
bq. Class IncrementMutation which inherits from Increment and Mutation
I think you will find some method signatures between them incompatible.
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498229#comment-13498229 ]
Ted Yu commented on HBASE-7093:
-------------------------------
{code}
for (Mutation m : mutations) {
...
+ advancedMemstoreWrite = region.prepareAppend((Append)m, now, !advancedMemstoreWrite);
{code}
Since prepareAppend() is called in a loop, if there're multiple Append's in mutations, the value of waitForPreviousMemstoreWrite would toggle between calls to prepareAppend(), right ?
That doesn't seem to be what you wanted.
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497770#comment-13497770 ]
Varun Sharma commented on HBASE-7093:
-------------------------------------
Hi, I have attached a sketch of a patch for supporting append operation (Increment seems more complicated) as part of Puts + Deletes. I am yet to add this to the miniBatch... function and also add tests. Needed to get any early input. Could someone please have a look ? Thanks !
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498210#comment-13498210 ]
Varun Sharma commented on HBASE-7093:
-------------------------------------
Hi Ted,
Thanks for taking a look.
HBASE 4583 forced appends and increments to wait for previous Memstore transactions to complete since otherwise we could have potential race conditions with puts into that same column.
However, if we allow Append(s) as part of MutateRow - this variable essentially allows us to do this "wait for previous memstore" transaction thing exactly once. So, we pass to prepareAppend whether we have historically advanced the memstore, if not, then prepareAppend shall advance it for us. Maybe i should rename the argument and variable in the process() call to be - "waitForPreviousMemstoreWrite" and the local variable to be advancedMemstoreWrite...
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498227#comment-13498227 ]
Andrew Purtell commented on HBASE-7093:
---------------------------------------
I think you mean HBASE-7114.
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498271#comment-13498271 ]
Varun Sharma commented on HBASE-7093:
-------------------------------------
Yeah, thats what I meant, I will send out a v2 with the tests. THanks !
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497778#comment-13497778 ]
Ted Yu commented on HBASE-7093:
-------------------------------
In javadoc, describe what the method does:
{code}
+ boolean prepareAppend(Append append, long now, boolean advanceMemstoreRead) throws IOException {
{code}
Also, add @return javadoc.
{code}
+ boolean advancedMemstoreRead = false;
+ if (advanceMemstoreRead) {
{code}
Why introducing a local variable which is spelled almost exactly the same as method parameter ?
Looking at the method's return value, it would be the same as that of the input parameter, advanceMemstoreRead. Looks like you don't need advancedMemstoreRead.
{code}
+ advancedMemstoreWrite = region.prepareAppend((Append)m, now, !advancedMemstoreWrite);
{code}
Can you explain the role for advancedMemstoreWrite above (especially the assignment) ?
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Varun Sharma updated HBASE-7093:
--------------------------------
Attachment: 7093-v1.txt
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498221#comment-13498221 ]
Varun Sharma commented on HBASE-7093:
-------------------------------------
Hi Andrew,
I remember seeing your other JIRA which talks about the incompatibility b/w Increment and Mutation - I can't find it now but it looks like doing Increment would be harder so i might not cover that in this JIRA (only cover Append here).
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498264#comment-13498264 ]
Varun Sharma commented on HBASE-7093:
-------------------------------------
Good catch - I will fix that part - anything else that might seem structurally wrong with the patch (it somewhat changes the RowProcessor interface) - I can fix this and also add tests in a subsequent patch
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Varun Sharma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Varun Sharma reassigned HBASE-7093:
-----------------------------------
Assignee: Varun Sharma
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7093) Couple Increments/Appends with
Put/Delete(s)
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498267#comment-13498267 ]
Ted Yu commented on HBASE-7093:
-------------------------------
bq. also add tests in a subsequent patch
Can you add test(s) in this JIRA ?
> Couple Increments/Appends with Put/Delete(s)
> --------------------------------------------
>
> Key: HBASE-7093
> URL: https://issues.apache.org/jira/browse/HBASE-7093
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.96.0
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Attachments: 7093-v1.txt
>
>
> See related issue - https://issues.apache.org/jira/browse/HBASE-4583
> Currently, we cannot bundle increment/append with put/delete operations. The above JIRA MVCC'izes the increment/append operations.
> One issue is that increment(s)/append(s) are not idempotent and hence repeating the transaction has an associated issue of leading to incorrect value/append results. This could be solved by passing additional tokens as part of the append(s).
> One possible high level approach could be:
> 1) Class IncrementMutation which inherits from Increment and Mutation
> 2) In the mutateRow call, we add a case for "IncrementMutation" object
> 3) Factor out the code wrapped inside the "lock and MVCC" from increment() function to internalIncrement.
> 4) Call internalIncrement from mutateRow and increment()
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira