You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2010/11/21 20:45:13 UTC
[jira] Created: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Coprocessors: Extend server side integration API to include HLog operations
---------------------------------------------------------------------------
Key: HBASE-3257
URL: https://issues.apache.org/jira/browse/HBASE-3257
Project: HBase
Issue Type: Sub-task
Reporter: Andrew Purtell
Assignee: Mingjie Lai
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3257:
----------------------------------
Attachment: 3257-3.patch
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
> Attachments: 3257-3.patch
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3257:
----------------------------------
Description:
Coprocessor based extensions should be able to:
* Observe, rewrite, or skip WALEdits as they are being written to the WAL
* Write arbitrary content into WALEdits
* Act on contents of WALEdits in the master context when splitting
* Act on contents of WALEdits in the regionserver context during reconstruction
One idea we had regards how CPs injecting arbitrary content into the WAL can play nicely with the master and replication is as follows: For replication we have already introduced the concept of scoping for WAL edits. This is expressed as an integer. Currently if scope=0, then the WALEdit is processed by the master and regionserver but not replicated; and if scope=1, then the edit is processed *and* replicated. We can change the semantics slightly to:
* If scope=0, the edit is split by the master into reconstruction logs but ignored by the base regionserver logic (_ignore scope_)
* If scope=1, the edit is processed by master and regionserver as normal (_default scope_)
* If scope=2, the edit is processed and replicated (_replication scope_)
Then CPs can stuff arbitrary KVs into a WALEdit, set scope=0, and pick up whatever was stored on the other side of reconstruction, but otherwise the data is simply passed through everywhere.
Fix Version/s: 0.92.0
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the master context when splitting
> * Act on contents of WALEdits in the regionserver context during reconstruction
> One idea we had regards how CPs injecting arbitrary content into the WAL can play nicely with the master and replication is as follows: For replication we have already introduced the concept of scoping for WAL edits. This is expressed as an integer. Currently if scope=0, then the WALEdit is processed by the master and regionserver but not replicated; and if scope=1, then the edit is processed *and* replicated. We can change the semantics slightly to:
> * If scope=0, the edit is split by the master into reconstruction logs but ignored by the base regionserver logic (_ignore scope_)
> * If scope=1, the edit is processed by master and regionserver as normal (_default scope_)
> * If scope=2, the edit is processed and replicated (_replication scope_)
> Then CPs can stuff arbitrary KVs into a WALEdit, set scope=0, and pick up whatever was stored on the other side of reconstruction, but otherwise the data is simply passed through everywhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Mingjie Lai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989363#comment-12989363 ]
Mingjie Lai commented on HBASE-3257:
------------------------------------
Patch was put to review board, but it's not forwarded to jira.
Here is the review request:
https://review.cloudera.org/r/1515/
Summary:
Coprocessors: Extend server side integration API to include HLog operations
Coprocessor based extensions should be able to:
- Observe, rewrite, or skip WALEdits as they are being written to the WAL
- Write arbitrary content into WALEdits
- Act on contents of WALEdits in the regionserver context during reconstruction
Code changes:
- a new coprocessor interface WALCPObserver is added which provides preWALWrite() and postWALWrite() upcalls to HLog, before and after HLog.write.append(), at doWrite().
- added 2 new upcalls for RegionObserver for WAL replay, preWALRestore() and postWALRestore().
- a sample implementation -- SampleRegionWALObserver -- was create which can add, remove, modify WALEdit before writing it to WAL.
- test cases which use SampleRegionWALObserver to test WAL write and replay.
- added coprocessor loading at TestHLog to make sure it doesn't affect HLog.
I need feedback for:
- The new cp interface name -- WALCPObserver -- is not perfect. The ideal name is WALObserver but it's been used already. Other options include HLogObserver(H is not preferred), LogObserver(confusion).
- No support for monitor master log splitting. I don't have a use case to support the requirement right now.
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3257:
----------------------------------
Status: Patch Available (was: Open)
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992013#comment-12992013 ]
stack commented on HBASE-3257:
------------------------------
Sorry I didn't get to a review. This is a great addition.
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
> Attachments: 3257-3.patch
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Mingjie Lai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingjie Lai updated HBASE-3257:
-------------------------------
Description:
Coprocessor based extensions should be able to:
* Observe, rewrite, or skip WALEdits as they are being written to the WAL
* Write arbitrary content into WALEdits
* Act on contents of WALEdits in the regionserver context during reconstruction
(update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
was:
Coprocessor based extensions should be able to:
* Observe, rewrite, or skip WALEdits as they are being written to the WAL
* Write arbitrary content into WALEdits
* Act on contents of WALEdits in the master context when splitting
* Act on contents of WALEdits in the regionserver context during reconstruction
One idea we had regards how CPs injecting arbitrary content into the WAL can play nicely with the master and replication is as follows: For replication we have already introduced the concept of scoping for WAL edits. This is expressed as an integer. Currently if scope=0, then the WALEdit is processed by the master and regionserver but not replicated; and if scope=1, then the edit is processed *and* replicated. We can change the semantics slightly to:
* If scope=0, the edit is split by the master into reconstruction logs but ignored by the base regionserver logic (_ignore scope_)
* If scope=1, the edit is processed by master and regionserver as normal (_default scope_)
* If scope=2, the edit is processed and replicated (_replication scope_)
Then CPs can stuff arbitrary KVs into a WALEdit, set scope=0, and pick up whatever was stored on the other side of reconstruction, but otherwise the data is simply passed through everywhere.
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992022#comment-12992022 ]
stack commented on HBASE-3257:
------------------------------
Did a quick review. +1 on commit. There are a few things you might consider in my remarks. The one that is a little worrying is notion that CP Host is instantiated even if CP is not configured on (This may just be me misreading code).
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
> Attachments: 3257-3.patch
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991802#comment-12991802 ]
Hudson commented on HBASE-3257:
-------------------------------
Integrated in HBase-TRUNK #1737 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1737/])
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
> Attachments: 3257-3.patch
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HBASE-3257) Coprocessors: Extend server side
integration API to include HLog operations
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3257:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk after a week on RB. Tests pass locally.
> Coprocessors: Extend server side integration API to include HLog operations
> ---------------------------------------------------------------------------
>
> Key: HBASE-3257
> URL: https://issues.apache.org/jira/browse/HBASE-3257
> Project: HBase
> Issue Type: Sub-task
> Components: coprocessors
> Reporter: Andrew Purtell
> Assignee: Mingjie Lai
> Fix For: 0.92.0
>
> Attachments: 3257-3.patch
>
>
> Coprocessor based extensions should be able to:
> * Observe, rewrite, or skip WALEdits as they are being written to the WAL
> * Write arbitrary content into WALEdits
> * Act on contents of WALEdits in the regionserver context during reconstruction
> (update: remove WALEdit monitoring at master split. No need to use scope to control the behavior here, since it could be done by a coprocessor implementation.)
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira