You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/04/17 19:41:21 UTC
[jira] Created: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Still a 'hole' in scanners, even after HBASE-532
------------------------------------------------
Key: HBASE-588
URL: https://issues.apache.org/jira/browse/HBASE-588
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Priority: Blocker
Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-588:
--------------------------------
Fix Version/s: 0.1.2
Add as blocker for 0.1.2
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.2.0, 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588-v6.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to BRANCH and TRUNK
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.2.0, 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588-v6.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-588) Still a 'hole' in scanners, even
after HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591748#action_12591748 ]
stack commented on HBASE-588:
-----------------------------
Committed to TRUNK after making Jim suggested changes.
Working on backport.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588-v6.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588.patch
First attempt at fix for this issue. Doesn't yet pass all tests so needs more work.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588.patch
Adding getNextRow and getFull methods that take foreground and background caches -- e.g. memcache and snapshot -- and amending the MemcacheScanner so it takes foreground and background caches to use in its calls to getNextRow and getFull
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Status: Patch Available (was: Open)
Please review
Local tests pass (Running verification that deadlock has been fixed)
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Fix Version/s: 0.1.2
The Jim soln. means keeping around possibly two Maps where one is at least 64M per columnfamily, per region while a scanner is outstanding.
Making this issue a fix for 0.1.2.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-588) Still a 'hole' in scanners, even
after HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590136#action_12590136 ]
stack commented on HBASE-588:
-----------------------------
Thinking on the Jim suggestion more, could give a new Scanner both the current memcache and the snapshot; would make it so Scanner was seeing current data at least until flush did a new snapshot (so still unpredictable staleness). We're not iterating any more so should be fine if the current memcache gets new updates while its being used by the Scanner. Would mean each Scanner would hold references to two pretty big Maps. Would change getFull so I could pass in the 'memcache' and 'snapshot' to use.
This might be the way to go for the branch fix since its easy (though unpredictable staleness issue). Would do the eventing for TRUNK.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-588) Still a 'hole' in scanners, even
after HBASE-532
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591740#action_12591740 ]
Jim Kellerman commented on HBASE-588:
-------------------------------------
NotServingRegionException: only whitespace differences
HStore: fix indentation in completeCompaction
Otherwise +1.
I'll pick up the changes to HRegion and HRegionServer and the Flusher in my patch for HBASE-572 so those will already be back-ported.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588-v6.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588-v4.patch
Version that goes against latest TRUNK.
Failed over on cluster. Need to look into it in morning. Also fails doing TTI. Will look at that also.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588-v2.patch
Uploading again with different name (added v2). This patch adds Observers who are notified when Store Readers change.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-588:
--------------------------------
Fix Version/s: (was: 0.1.2)
0.2.0
Assignee: stack
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588-v6.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: (was: 588.patch)
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588-v5.patch
Add in fix for this -- though not caused by this patch. Also some clean up in Flusher added. Please review.
{code}
Java stack information for the threads listed above:
===================================================
"IPC Server handler 3 on 60020":
at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:118)
- waiting to lock <0xb6526348> (a java.util.HashSet)
at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1527)
- locked <0xb66c0708> (a java.lang.Integer)
at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1318)
at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1101)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher":
at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:987)
- waiting to lock <0xb66c0708> (a java.lang.Integer)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:923)
at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:171)
- locked <0xb6526348> (a java.util.HashSet)
at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:94)
Found 1 deadlock.
{code}
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588-v6.patch
Moved flush request out from under update lock; no need to be inside the lock scope.
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588-v4.patch, 588-v5.patch, 588-v6.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-588) Still a 'hole' in scanners, even after
HBASE-532
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-588:
------------------------
Attachment: 588-v3.patch
{code}
Add a ChangedReadersObserver interface. HStore notifies registered
observers when list of HStoreFile Readers changes -- at flush time
and at compaction time. Scanners are only current observers.
M src/test/org/apache/hadoop/hbase/MultiRegionTable.java
(getCacheFlushListern): Renamed as getFlushRequester
M src/java/org/apache/hadoop/hbase/regionserver/HStore.java
Added synchronizations on this.storefiles.
(activeScanners, newScannerLock): Removed. We no longer try to
block out scanners when compacting (Turns out same functionality
for blocking scanners is up in HRegion used around closing/split).
(changedReadersObservers): Added.
(updateReaders): New method that manages the insertion of new
reader on flush. Also calls new notifyChangedReadersObservers.
(notifyChagnedReadersObservers, addChangedReadersObserver,
deleteChangedReadersObservers): Added.
(completeCompaction): Previous deleting old store files, we'd
remove from this.storefiles and delete all in one step; now we
do the remove first, notify all observers of readers, and then
do the delete so observers have a chance to clean up any old
references to files about to be deleted. Removed all the lockout
of new scanner creation and wait on old scanners to come int.
(updateActiveScanners): Removed.
(getStorefiles): Accessor. Added.
M src/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
Added implementation of new ChangedReadersObserver interface.
Added a lock that we hold when 'nexting' and when changing the
set of readers out from under the scanner.
Changed the constructor moving bulk into new openReaders method
that we reuse when list of Readers changes.
(next): Moved head of this method into new getNextViableRow (used
to be called chosenRow and chosenTimestamp). New method returns
simple datastructure of row and timestamp (ViableRow).
(close): On close, remove ourselves as ChangedReadersObserver (we
added ourselves in the constructor).
(updateReaders): Changed the set of Readers out from under the
Scanner.
A src/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
Added.
M src/java/org/apache/hadoop/hbase/regionserver/Flusher.java
Changed name of the interface we implement from CacheFlushListener to
FlushRequester.
D src/java/org/apache/hadoop/hbase/regionserver/CacheFlushListener.java
Renamed as FlushRequester.
M src/java/org/apache/hadoop/hbase/regionserver/HStoreScanner.java
Remove update of activeScanners.
A src/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
Added. Rename of CacheFlushListener.
M src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Renamed method getCacheFlushListener as getFlushRequester.
M src/java/org/apache/hadoop/hbase/regionserver/HAbstractScanner.java
Formatting.
M src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
CacheFlushListener was renamed as FlushListener.
{code}
> Still a 'hole' in scanners, even after HBASE-532
> ------------------------------------------------
>
> Key: HBASE-588
> URL: https://issues.apache.org/jira/browse/HBASE-588
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.2
>
> Attachments: 588-v2.patch, 588-v3.patch, 588.patch
>
>
> Before HBASE-532, as soon as a flush started, we called snapshot. Snapshot used to copy current live memcache into a 'snapshot' TreeMap inside in Memcache. This snapshot TreeMap was an accumulation of all snapshots since last flush. Whenever we took out a scanner, we'd do a copy of this snapshot into a new backing map carried by the scanner (Every outstanding Scanner had complete copy). Memcache snapshots were cleared when a flush started. Flushing could take near no time to up to tens of seconds during which an scanners taken out meantime would not see the edits in the snapshot currently being flushed and gets or getFull would also return incorrect answers because the content of the snapshot was not available to them.
> HBASE-532 made it so the snapshot was available until flush was done -- until a file had made it out to disk. This fixed gets and getFull and any scanners taken out during flushing. But there is still a hole. Any outstanding scanners will be going against the state of Store Readers at time scanner was opened; they will not see the new flush file.
> Chatting about this on IRC, Jim suggests that we pass either memcache or current snapshot to each Scanner (Pass the snapshot if not empty). The notion is that the Scanner would hold on to the Scanner reference should it be cleared by flushing. Upside is that scanner wouldn't have to be concerned with the new flush that has been put out to disk. Downsides are that Scanner data could be way stale if for instance the memcache was near to flushing but we hadn't done it yet. And we wouldn't be clearing the snapshot promptly so would be some memory pressure.
> Another suggestion is that flushing send an event. Listeners such as outstanding scanners would notice event and open the new Reader. Would have to skip forward in the new Reader to catch up with the current set but shouldn't be bad. Same mechanism could be used to let compactions be moved into place while scanners were outstanding closing down all existing readers skipping to the current 'next' location in the new compacted store file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.