You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Akash Ashok (Created) (JIRA)" <ji...@apache.org> on 2011/10/02 07:08:34 UTC
[jira] [Created] (HBASE-4529) Make WAL Pluggable
Make WAL Pluggable
-------------------
Key: HBASE-4529
URL: https://issues.apache.org/jira/browse/HBASE-4529
Project: HBase
Issue Type: Improvement
Components: regionserver, wal
Reporter: Akash Ashok
Make WAL a pluggable Component. Thus making it easier to configure to write to different filesystem and even multiple filesystems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4529) Make WAL Pluggable
Posted by "Ted Yu (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4529:
--------------------------
Description:
Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
>From Stack:
Pluggable WAL component would need to check that the split can deal w/
multiple logs written by the one server concurrently (sort by sequence
edit id after sorting on all the rest that makes up a wal log key).
>From Jesse Yates:
It would be nice to be able to tie pluggable WAL component into a service that logs directly to
disk, rather than go through HDFS giving some potentially awesome speedup at
the cost of having to write a logging service that handles replication, etc.
>From Karthik Tunga:
Along with the log replaying part, logic is also needed for log roll.
This, I think, is easier compared to the merging of the logs. Any edits less
than the last sequence number on the file system can be removed from all
the WALs.
was:Make WAL a pluggable Component. Thus making it easier to configure to write to different filesystem and even multiple filesystems.
> Make WAL Pluggable
> -------------------
>
> Key: HBASE-4529
> URL: https://issues.apache.org/jira/browse/HBASE-4529
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
> Reporter: Akash Ashok
> Labels: regionserver, wal
>
> Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
> From Stack:
> Pluggable WAL component would need to check that the split can deal w/
> multiple logs written by the one server concurrently (sort by sequence
> edit id after sorting on all the rest that makes up a wal log key).
> From Jesse Yates:
> It would be nice to be able to tie pluggable WAL component into a service that logs directly to
> disk, rather than go through HDFS giving some potentially awesome speedup at
> the cost of having to write a logging service that handles replication, etc.
> From Karthik Tunga:
> Along with the log replaying part, logic is also needed for log roll.
> This, I think, is easier compared to the merging of the logs. Any edits less
> than the last sequence number on the file system can be removed from all
> the WALs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4529) Make WAL Pluggable
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178468#comment-13178468 ]
dhruba borthakur commented on HBASE-4529:
-----------------------------------------
Akash: I am very much interested in this feature. Just curious: are you interested in working on it?
> Make WAL Pluggable
> -------------------
>
> Key: HBASE-4529
> URL: https://issues.apache.org/jira/browse/HBASE-4529
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
> Reporter: Akash Ashok
> Labels: regionserver, wal
>
> Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
> From Stack:
> Pluggable WAL component would need to check that the split can deal w/
> multiple logs written by the one server concurrently (sort by sequence
> edit id after sorting on all the rest that makes up a wal log key).
> From Jesse Yates:
> It would be nice to be able to tie pluggable WAL component into a service that logs directly to
> disk, rather than go through HDFS giving some potentially awesome speedup at
> the cost of having to write a logging service that handles replication, etc.
> From Karthik Tunga:
> Along with the log replaying part, logic is also needed for log roll.
> This, I think, is easier compared to the merging of the logs. Any edits less
> than the last sequence number on the file system can be removed from all
> the WALs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4529) Make WAL Pluggable
Posted by "Akash Ashok (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akash Ashok reassigned HBASE-4529:
----------------------------------
Assignee: (was: Akash Ashok)
> Make WAL Pluggable
> -------------------
>
> Key: HBASE-4529
> URL: https://issues.apache.org/jira/browse/HBASE-4529
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
> Reporter: Akash Ashok
> Labels: regionserver, wal
>
> Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
> From Stack:
> Pluggable WAL component would need to check that the split can deal w/
> multiple logs written by the one server concurrently (sort by sequence
> edit id after sorting on all the rest that makes up a wal log key).
> From Jesse Yates:
> It would be nice to be able to tie pluggable WAL component into a service that logs directly to
> disk, rather than go through HDFS giving some potentially awesome speedup at
> the cost of having to write a logging service that handles replication, etc.
> From Karthik Tunga:
> Along with the log replaying part, logic is also needed for log roll.
> This, I think, is easier compared to the merging of the logs. Any edits less
> than the last sequence number on the file system can be removed from all
> the WALs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4529) Make WAL Pluggable
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176459#comment-13176459 ]
dhruba borthakur commented on HBASE-4529:
-----------------------------------------
This will be a great feature!
HDFS provides io-fencing for HBase log. When a regionserver dies, the master renames the log directory so that the oldregionserver cannot continue to write any new transactions to the transaction log. (HDFS provides an api that fails a file-creation-request if an intermediate path component is non-existant). It will be nice to get the pluggable-wal API support this type of operation.
> Make WAL Pluggable
> -------------------
>
> Key: HBASE-4529
> URL: https://issues.apache.org/jira/browse/HBASE-4529
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
> Reporter: Akash Ashok
> Assignee: Akash Ashok
> Labels: regionserver, wal
>
> Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
> From Stack:
> Pluggable WAL component would need to check that the split can deal w/
> multiple logs written by the one server concurrently (sort by sequence
> edit id after sorting on all the rest that makes up a wal log key).
> From Jesse Yates:
> It would be nice to be able to tie pluggable WAL component into a service that logs directly to
> disk, rather than go through HDFS giving some potentially awesome speedup at
> the cost of having to write a logging service that handles replication, etc.
> From Karthik Tunga:
> Along with the log replaying part, logic is also needed for log roll.
> This, I think, is easier compared to the merging of the logs. Any edits less
> than the last sequence number on the file system can be removed from all
> the WALs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4529) Make WAL Pluggable
Posted by "Akash Ashok (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akash Ashok reassigned HBASE-4529:
----------------------------------
Assignee: Akash Ashok
> Make WAL Pluggable
> -------------------
>
> Key: HBASE-4529
> URL: https://issues.apache.org/jira/browse/HBASE-4529
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
> Reporter: Akash Ashok
> Assignee: Akash Ashok
> Labels: regionserver, wal
>
> Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
> From Stack:
> Pluggable WAL component would need to check that the split can deal w/
> multiple logs written by the one server concurrently (sort by sequence
> edit id after sorting on all the rest that makes up a wal log key).
> From Jesse Yates:
> It would be nice to be able to tie pluggable WAL component into a service that logs directly to
> disk, rather than go through HDFS giving some potentially awesome speedup at
> the cost of having to write a logging service that handles replication, etc.
> From Karthik Tunga:
> Along with the log replaying part, logic is also needed for log roll.
> This, I think, is easier compared to the merging of the logs. Any edits less
> than the last sequence number on the file system can be removed from all
> the WALs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4529) Make WAL Pluggable
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178472#comment-13178472 ]
Zhihong Yu commented on HBASE-4529:
-----------------------------------
@Dhruba:
Akash is working on HBASE-4224 at the moment.
Feel free to give this nice feature a try.
> Make WAL Pluggable
> -------------------
>
> Key: HBASE-4529
> URL: https://issues.apache.org/jira/browse/HBASE-4529
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, wal
> Reporter: Akash Ashok
> Labels: regionserver, wal
>
> Make WAL a pluggable, configurable component, thus making it easier to write to different filesystems (including multiple filesystems).
> From Stack:
> Pluggable WAL component would need to check that the split can deal w/
> multiple logs written by the one server concurrently (sort by sequence
> edit id after sorting on all the rest that makes up a wal log key).
> From Jesse Yates:
> It would be nice to be able to tie pluggable WAL component into a service that logs directly to
> disk, rather than go through HDFS giving some potentially awesome speedup at
> the cost of having to write a logging service that handles replication, etc.
> From Karthik Tunga:
> Along with the log replaying part, logic is also needed for log roll.
> This, I think, is easier compared to the merging of the logs. Any edits less
> than the last sequence number on the file system can be removed from all
> the WALs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira