You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2009/06/22 19:28:07 UTC
[jira] Created: (HBASE-1563) incrementColumnValue does not write to
WAL
incrementColumnValue does not write to WAL
------------------------------------------
Key: HBASE-1563
URL: https://issues.apache.org/jira/browse/HBASE-1563
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Fix For: 0.20.0
Attachments: HBASE-1563-v1.patch
Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-1563) incrementColumnValue does not
write to WAL
Posted by Ryan Rawson <ry...@gmail.com>.
The old code did a row lock and was fast
On Jun 23, 2009 4:03 PM, "stack (JIRA)" <ji...@apache.org> wrote:
[
https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723344#action_12723344]
stack commented on HBASE-1563: ------------------------------
Patch looks good except for the identified weirdness where we write to WAL
after the update. To do this properly you'd need to row lock and make a
copy. Is that too onerous a price to pay?
> incrementColumnValue does not write to WAL >
------------------------------------------ > > ...
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch,
HBASE-1563-v3.patch
> > > Incrementing never writes to the WAL. Under failure scenarios, you
will lose all increments s...
Re: [jira] Updated: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by Ryan Rawson <ry...@gmail.com>.
we're going to have to rethink the inplace increment. That causes a
potential race. I want to write a bench suite to try to replicate it.
On Jun 23, 2009 3:45 PM, "Jonathan Gray (JIRA)" <ji...@apache.org> wrote:
[
https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issue.
..
Attachment: HBASE-1563-v3.patch
- Removes unnecessary Get in client code
- Removed duplicate server calls in the client code, default just calls full
w/ true
- Replaced increment javadoc with "Atomically increments a column value. If
the column value already exists and is not a big-endian long, this could
throw an exception." Is that more clear?
@Ryan I got most the way through splitting up Store.incrementColumnValue
logic so we figure out what we have to do first, then write to wal, then
perform the insert. The issue is when we increment in-place in the
memcache. If we truly don't want to write until the wal append, we'd have
to make a copy of the KV, perform the increment on it, and then either
increment the original memcache value or swap. This more or less negates
the in-place increment optimization. Thoughts?
> incrementColumnValue does not write to WAL >
------------------------------------------ > > ...
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch,
HBASE-1563-v3.patch
> > > Incrementing never writes to the WAL. Under failure scenarios, you
will lose all increments s...
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723389#action_12723389 ]
Jonathan Gray commented on HBASE-1563:
--------------------------------------
We are already getting a row lock, there is no race condition that I can see.
I'd really like the increment-in-place over making a new KV each time we increment. The downside is if the regionserver dies in the brief window after the update and before the WAL append, we lose a single increment. If it dies on either side, then nothing would change.
The upside is we don't allocate a new KV each time. That's what I'm after with increments, doing them in place and not doing a new insertion each time (and new allocation). I have an internal ID assignment system that uses custom patches on 0.19. Was hoping to drop our custom patches and use this out of the box, but if we copy the KV each time I'll likely patch it to not do that :) I'd rather not lose any increments but the window is very small and you lose at most a single increment. If a KV is 100 bytes, and I increment just 1M times in a day, i've unnecessarily allocated 100MB (and the worst kind, lots of small 100 byte allocations that stick around for a bit).
We still satisfy the property of "Once the increment has returned successfully, it is safe". This weird scenario would actually only happen in a case where the increment started but never returned (so client should not assume it was successful anyways).
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Attachment: HBASE-1563-v1.patch
This patch adds a writeToWAL argument and appends to the edit log on every increment, however the boolean is not exposed to the client in this patch and is default true. Wanted to discuss with others before going further.
Thoughts?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1563.
--------------------------
Resolution: Fixed
Release Note: Adds new API with flag to incrementColumnValue for whether or not to write WAL
Hadoop Flags: [Reviewed]
Thanks for the patch Jon
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch, HBASE-1563-v4.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723454#action_12723454 ]
Jonathan Gray commented on HBASE-1563:
--------------------------------------
For now, let's just make the copy and make sure things are safe. In addition to 1577. Good stuff ryan.
Stack, you've brought your improper usage of /me onto the web now!
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Status: Patch Available (was: In Progress)
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723395#action_12723395 ]
Andrew Purtell commented on HBASE-1563:
---------------------------------------
bq. We still satisfy the property of "Once the increment has returned successfully, it is safe". This weird scenario would actually only happen in a case where the increment started but never returned (so client should not assume it was successful anyways).
This makes sense to me.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723397#action_12723397 ]
ryan rawson commented on HBASE-1563:
------------------------------------
a row lock is only for writing. The race happens when someone is reading the value you are incrementing. They could copy a partially updated view into the RPC buffer, and the user could see something odd and unexpected (values going down for example).
If we used a ConcurrentSkipListSet, we can't put a duplicate, which requires us to temporarily remove the KeyValue, then put it back, opens a hole whereby we can look for a value and it not be there.
So to support not-in-place modification we need to also move to a ConcurrentSkipListMap in memcache.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723415#action_12723415 ]
stack commented on HBASE-1563:
------------------------------
/me man, i love OSS
Someone please review 1577 -- it converts memcache to CSLM from CSLS.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Attachment: HBASE-1563-v2.patch
Please review.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723359#action_12723359 ]
stack commented on HBASE-1563:
------------------------------
Why is there a race if a row lock is outstanding Ryan?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Irfan Mohammed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724086#action_12724086 ]
Irfan Mohammed commented on HBASE-1563:
---------------------------------------
Can we please ensure that the "atomic" puts are honored?
I migrated lot of my code to use the incrementColumnValue() because of the atomic nature of this call and it really reduces the data footprint and also the query time when aggregating the cell values.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch, HBASE-1563-v4.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Status: Patch Available (was: Open)
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Attachment: HBASE-1563-v3.patch
- Removes unnecessary Get in client code
- Removed duplicate server calls in the client code, default just calls full w/ true
- Replaced increment javadoc with "Atomically increments a column value. If the column value already exists and is not a big-endian long, this could throw an exception." Is that more clear?
@Ryan I got most the way through splitting up Store.incrementColumnValue logic so we figure out what we have to do first, then write to wal, then perform the insert. The issue is when we increment in-place in the memcache. If we truly don't want to write until the wal append, we'd have to make a copy of the KV, perform the increment on it, and then either increment the original memcache value or swap. This more or less negates the in-place increment optimization. Thoughts?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722704#action_12722704 ]
Jonathan Gray commented on HBASE-1563:
--------------------------------------
+1 patch coming...
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722796#action_12722796 ]
ryan rawson commented on HBASE-1563:
------------------------------------
2 things:
- in the client code you build a 'get' that isnt used.
- we are writing to the WAL _after_ the increment, this may be the only way to do it effectively, but we should also think about this carefully.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Status: In Progress (was: Patch Available)
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724235#action_12724235 ]
Jonathan Gray commented on HBASE-1563:
--------------------------------------
@Irfan Excuse me... Got confused for a second. Once HBASE-1577 goes in, these absolutely *will* have atomic behavior. Each is guaranteed to have worked and performed the increment once it returns, and all readers will always see the latest value.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch, HBASE-1563-v4.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Attachment: HBASE-1563-v4.patch
New patch that copies the existing KV instead of incrementing in-place.
This is now broken. The CSLMap has the same behavior as the CSLSet. Since the keys are considered "equal" the values might be updated in the map (in this case meaningless) but they don't swap the key with the new one (since it's equal). It does not have the behavior we wanted, an atomic put that replaces the Key in case of collisions.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch, HBASE-1563-v4.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723344#action_12723344 ]
stack commented on HBASE-1563:
------------------------------
Patch looks good except for the identified weirdness where we write to WAL after the update. To do this properly you'd need to row lock and make a copy. Is that too onerous a price to pay?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722841#action_12722841 ]
Erik Holstad commented on HBASE-1563:
-------------------------------------
@Ryan I think the Get is just some old leftover when it was used to be sent over that way and should now be removed.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722702#action_12722702 ]
Andrew Purtell commented on HBASE-1563:
---------------------------------------
IMHO, it's fine to expose the boolean to the client but there should be a client side API alternative which does not expose the boolean which internally sets writeToWAL as true. Everybody will be happy. Let's just do it.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1563) incrementColumnValue does not write to
WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1563:
---------------------------------
Status: In Progress (was: Patch Available)
Will post patch tomorrow
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722751#action_12722751 ]
Andrew Purtell commented on HBASE-1563:
---------------------------------------
Code is fine.
This portion of the javadoc:
bq. If the column value isn't long-like, this could throw an exception.
I understand what you are saying, but I think this may just confuse users. Can you consider an alternate wording?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724226#action_12724226 ]
Jonathan Gray commented on HBASE-1563:
--------------------------------------
@Irfan First and foremost, we need HBASE-1577, but that is looking good. Beyond that, we need to think more about how to make this behave atomically. In thinking about it now, I do see some potential issues/race conditions that will lead to missing increments (with HBASE-1577 things will always _work_ but does not ensure atomicity of the increments). Need to think more.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch, HBASE-1563-v4.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722750#action_12722750 ]
stack commented on HBASE-1563:
------------------------------
Why not have old incrementColumnValue call the new one with a flag set to 'true'?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723356#action_12723356 ]
stack commented on HBASE-1563:
------------------------------
Ryan notes that old code did row lock and was fast.
So, we should go ahead and pull the KV from memcache, copy it, increment it, add to WAL, then insert back into memcache? It won't be too slow? Maybe only do this if write to wal is enabled?
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1563) incrementColumnValue does not write
to WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723450#action_12723450 ]
stack commented on HBASE-1563:
------------------------------
So, interesting, need to operate on copy so readers don't see partially changed KV. That makes sense.
> incrementColumnValue does not write to WAL
> ------------------------------------------
>
> Key: HBASE-1563
> URL: https://issues.apache.org/jira/browse/HBASE-1563
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1563-v1.patch, HBASE-1563-v2.patch, HBASE-1563-v3.patch
>
>
> Incrementing never writes to the WAL. Under failure scenarios, you will lose all increments since the last flush.
> Do we want to expose the option to the client as to whether to write to WAL or not?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.