You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2011/09/24 02:57:26 UTC
[jira] [Created] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Ability for an application to store metadata into the transaction log
---------------------------------------------------------------------
Key: HBASE-4477
URL: https://issues.apache.org/jira/browse/HBASE-4477
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117791#comment-13117791 ]
Hudson commented on HBASE-4477:
-------------------------------
Integrated in HBase-0.92 #31 (See [https://builds.apache.org/job/HBase-0.92/31/])
HBASE-4477 Ability for an application to store metadata into the transaction log (dhruba via jgray)
jgray :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.0
>
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117446#comment-13117446 ]
Ted Yu commented on HBASE-4477:
-------------------------------
I second Todd's idea.
Should CPPutInfo be created in this JIRA and we can work on other parts in another JIRA ?
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113981#comment-13113981 ]
Ted Yu commented on HBASE-4477:
-------------------------------
{code}
+ public static final String WAL_META_ATTRIBUTE_NAME = "_hbaseWalMetadata";
{code}
Maybe the 'hbase' is not needed in the WAL_META_ATTRIBUTE_NAME.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114104#comment-13114104 ]
dhruba borthakur commented on HBASE-4477:
-----------------------------------------
This can be implemented by WALObserver coprocessor. But do you think we should keep this feature internal to our install or do you think we should contribute it back to Apache? If we can put it somewhere in open source (not necessarily in Apache, is github preferable?), we can also open source our cluster-replication software.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117449#comment-13117449 ]
Jonathan Gray commented on HBASE-4477:
--------------------------------------
+1 on CPPutInfo, CPGetInfo, etc...
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Andrew Purtell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116675#comment-13116675 ]
Andrew Purtell commented on HBASE-4477:
---------------------------------------
@dhruba
bq. can I change the signature of RegionObserver.prePut() to take in two additional arguments: a WALEdit and Put object
That sounds good. Since there is no release yet, no deprecation is necessary. Can be done simply by a patch on this issue I'd say.
@Jon
bq. If things are built only on Coprocessor interfaces, do people see us including these in some kind of coprocessor contrib or should they be out on github or something
I think it depends. Some stuff like security we'd clearly want to bundle. And by that not a resurrection of contrib, instead as another package in main/. Random additions that people build for themselves should go up on GitHub. My opinion is anything that is core to a group of use cases is a candidate for bundling, if a contributor or committer wants to maintain it, and/or if people generally feel it is a good idea to bring the candidate in.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117562#comment-13117562 ]
dhruba borthakur commented on HBASE-4477:
-----------------------------------------
> I'm curious if passing the cluster ID to internalPut and internalDelete is necessary?
I do not think that this is necessary. We can clean them up when we merge all the parameters to the coprocessor api.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4477:
------------------------------------
Attachment: hlogMetadata1.txt
The initial design I am considering here:
1. The Put call already has a map of arbitrary attributes (keys/values). The application will set a special key named "_hbaseWalMetadata". The value of this attribute is what needs to be stored in the HLog. From this perspective, the format of the Put call does not need any change.
2. The regionserver will store a metakv in the same Waledit record as the transaction. This kv will have column family name as ""METAFAMILY" and column name as "METACOL".
3. Code already exists to ignore kvs with column family as ""METAFAMILY" from the log splitting process. The reason is because we already write such records to the HLog (see completeCacheFlush)
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Andrew Purtell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117500#comment-13117500 ]
Andrew Purtell commented on HBASE-4477:
---------------------------------------
In my opinion prefixing anything regarding coprocessors with CP should be an anti-pattern.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117459#comment-13117459 ]
Todd Lipcon commented on HBASE-4477:
------------------------------------
Sure, I don't particularly care if it happens as part of this JIRA or separately. Just wanted to drop my 2 cents :)
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117451#comment-13117451 ]
dhruba borthakur commented on HBASE-4477:
-----------------------------------------
Hi Todd, I like your idea, and I will create another jira for that one. It will have to touch each and every coprocessor api. (If you are ok with it, I would like this jira to make the prePut and preGet be similar in the sense that they both take the Put/get object).
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116106#comment-13116106 ]
dhruba borthakur commented on HBASE-4477:
-----------------------------------------
@Andrew: I would like to do this via a co-processor API, can I change the signature of RegionObserver.prePut() to take in two additional arguments: a WALEdit and Put object? If so, shall I mark the existing prePut precessor api as Deprecated?
anybody else have any opinions on where to put the code that implements this functionality? One place is org.apache.hadoop.hbase.coprocessor.library.walMetadata.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Gary Helmling (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117472#comment-13117472 ]
Gary Helmling commented on HBASE-4477:
--------------------------------------
@Todd,
I like the idea of using XXXInfo objects to insulate coprocessors from API changes they're not using.
@Dhruba,
Agree that it would be better to tackle that in a separate JIRA and keep all of the APIs in sync. Adopting just PutInfo here would leave us with an oddly inconsistent API after this patch.
@All,
Can we drop the "CP" prefix and just use PutInfo, GetInfo, DeleteInfo, etc? It looks ugly to me and unnecessary if we place these classes under o.a.h.h.coprocessor for namespacing.
Also, I like that this makes the exposed Observer APIs less brittle, but it does add a fair amount of complexity to the APIs in new classes. To carry this through, we really need a XXXInfo class per operation...
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4477:
------------------------------------
Attachment: coprocessorPut1.txt
Implemented andrew's suggestion of enahncing the prePut, postPut, preDelete and postDelete apis to take in the Put/Delete object itself.
In the process of running tests.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117441#comment-13117441 ]
Todd Lipcon commented on HBASE-4477:
------------------------------------
Just a thought while we're changing the APIs: might make sense to do something like introduce the following interface:
{code}
interface CPPutInfo {
public Put getPutObject();
public boolean getWriteToWAL();
public WALEdit getWalEdit();
}
{code}
then, if we want to extend this API with more info about the put later, we can simply add more methods and not break compatibility? We should probably do this in another JIRA, and for all of the RegionObserver hooks - this JIRA just gave me the idea.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Andrew Purtell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-4477:
----------------------------------
Attachment: coprocessorPut2.txt
+1
Attached updated patch 'coprocessorPut2.txt" which removes some unused locals.
I'm curious if passing the cluster ID to internalPut and internalDelete is necessary? They can be retrieved from the Put or Delete, or the Put or Delete can be modified. I don't know enough about replication there: The put or delete gets written to the WAL with one UUID but internally contains another?
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117479#comment-13117479 ]
Ted Yu commented on HBASE-4477:
-------------------------------
Todd didn't mention which namespace CPPutInfo should be placed.
If we cannot foresee that other components would utilize PutInfo, we should place PutInfo in o.a.h.h.coprocessor without the CP prefix.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117496#comment-13117496 ]
Jonathan Gray commented on HBASE-4477:
--------------------------------------
PutInfo seems overly generic but I agree that CPPutInfo is straight ugly. And I keep thinking it says CPUInfo.
So Dhruba should just extend the API for now and we can introduce these new classes in a follow-up jira.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117428#comment-13117428 ]
dhruba borthakur commented on HBASE-4477:
-----------------------------------------
I ran all unit tests. All unit tests pass, except these: TestDistributedLogSplitting, TestRollingRestart, TestHTablePool. And these failures are not related to my patch.
Andrew: will you be able to review this patch, especially because it changes the coprocessor interface?
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117840#comment-13117840 ]
Hudson commented on HBASE-4477:
-------------------------------
Integrated in HBase-TRUNK #2270 (See [https://builds.apache.org/job/HBase-TRUNK/2270/])
HBASE-4477 Ability for an application to store metadata into the transaction log (dhruba via jgray)
jgray :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.0
>
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Gary Helmling (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117507#comment-13117507 ]
Gary Helmling commented on HBASE-4477:
--------------------------------------
+1 on coprocessorPut2.txt, which just removes the unused familyMap local vars in RegionCoprocessorHost.
I'm curious about the cluster ID question as well though. Don't know enough about the replication stuff to understand the current usage.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114003#comment-13114003 ]
Andrew Purtell commented on HBASE-4477:
---------------------------------------
We created the WALObserver coprocessor for this use case. You can build this simply and unobtrusively with a RegionObserver to get the attributes on the Put and so something with them, like add to the KV; and then a WALObserver can take application actions according to the added metadata or otherwise filter it out so it doesn't affect core.
So my question is why does this need to be in core? Is the WALObserver insufficient? Is this for <= 0.90?
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115006#comment-13115006 ]
Jonathan Gray commented on HBASE-4477:
--------------------------------------
I think this will take a few changes to the Coprocessor API if we want to only use RegionObserver and WALObserver.
Andy, what should one do if additional arguments are needed to, say, the prePut() call? Do we add multiple prePut() calls to the RegionObserver interface? Potentially deprecate the old ones?
If things are built only on Coprocessor interfaces, do people see us including these in some kind of coprocessor contrib or should they be out on github or something?
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Jonathan Gray (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-4477.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: Reviewed
Committed to trunk and 92 branch. Thanks for all the reviews and feedback all, nice stuff Dhruba. Dhruba opened HBASE-4514 for the follow-up discussed here.
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.0
>
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117497#comment-13117497 ]
Jonathan Gray commented on HBASE-4477:
--------------------------------------
And it looks like the patch from this morning does exactly that.
I'm +1 on coprocessorPut1.txt. Someone else want to review?
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4477) Ability for an application to store
metadata into the transaction log
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117450#comment-13117450 ]
Jonathan Gray commented on HBASE-4477:
--------------------------------------
And yeah, maybe introduce CPPutInfo in this JIRA and open a follow-up to change the others
> Ability for an application to store metadata into the transaction log
> ---------------------------------------------------------------------
>
> Key: HBASE-4477
> URL: https://issues.apache.org/jira/browse/HBASE-4477
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: coprocessorPut1.txt, hlogMetadata1.txt
>
>
> mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE.
> The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira