You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2009/06/17 22:28:08 UTC
[jira] Created: (HBASE-1535) Add client ability to perform
mutations without the WAL
Add client ability to perform mutations without the WAL
-------------------------------------------------------
Key: HBASE-1535
URL: https://issues.apache.org/jira/browse/HBASE-1535
Project: Hadoop HBase
Issue Type: New Feature
Affects Versions: 0.20.0
Reporter: Jonathan Gray
Fix For: 0.20.1
There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1535:
---------------------------------
Attachment: HBASE-1535-v2.patch
Adds serialization.
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1535-v1.patch, HBASE-1535-v2.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720983#action_12720983 ]
Jonathan Gray commented on HBASE-1535:
--------------------------------------
Bypassing the WAL breaks a whole bunch of stuff. It would need to come with a warning about what it means.
This "feature" doesn't break 1295, rather just like setting this makes it so you can't recover under RS failure, it also makes it so you are not replicating your edits.
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1535-v1.patch, HBASE-1535-v2.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-1535:
------------------------------------
Assignee: Jonathan Gray
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1535-v1.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1535.
--------------------------
Resolution: Fixed
Fix Version/s: (was: 0.20.1)
0.20.0
Hadoop Flags: [Reviewed]
Committed after adding in extra javadoc that no wal is dangerous.
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1535-v1.patch, HBASE-1535-v2.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1535:
---------------------------------
Attachment: HBASE-1535-v1.patch
Adds a writeToWAL boolean into Put. Setter/Getter of the same name.
Going to run an import test and see what this does.
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1535-v1.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721027#action_12721027 ]
Jim Kellerman commented on HBASE-1535:
--------------------------------------
There ought to be a neon flashing sign on this option.
As for bulk import, would you use that on an existing table that had data in it? You *could* get around the replication problem by shutting down the other replica(s) that are accepting a feed from the replica getting
the bulk update, then shut it down and copy the table(s) to the other HDFS(s).
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1535-v1.patch, HBASE-1535-v2.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720981#action_12720981 ]
Andrew Purtell commented on HBASE-1535:
---------------------------------------
If clients are bypassing the WAL for performance during bulk uploads, obviously that totally breaks HBASE-1295 as currently proposed.
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1535-v1.patch, HBASE-1535-v2.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1535) Add client ability to perform
mutations without the WAL
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720979#action_12720979 ]
Jonathan Gray commented on HBASE-1535:
--------------------------------------
Wowsa.
{noformat}
Import test, single client. 10 rows, each with 2M columns. Qualifiers are 8 bytes, values are 256 bytes.
[writeToWAL = true]
Inserted 10 rows each with 2000000 total columns in 792979ms (79297.9ms/row)
[writeToWAL = false]
Inserted 10 rows each with 2000000 total columns in 321308ms (32130.8ms/row)
{noformat}
Over 2X improvement in import speed.
> Add client ability to perform mutations without the WAL
> -------------------------------------------------------
>
> Key: HBASE-1535
> URL: https://issues.apache.org/jira/browse/HBASE-1535
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.1
>
> Attachments: HBASE-1535-v1.patch, HBASE-1535-v2.patch
>
>
> There are a number of cases where a client might not want/need its edits to be written into the HLog. The most obvious use of this would be during a bulk import where we want the best performance possible and data loss is acceptable (can rerun the import).
> Does this become a flag on Put or a special mode of HTable?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.