You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/07/07 19:44:52 UTC
[jira] Created: (HBASE-2821) Keep young storefiles at lower
replication
Keep young storefiles at lower replication
------------------------------------------
Key: HBASE-2821
URL: https://issues.apache.org/jira/browse/HBASE-2821
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Todd Lipcon
jgray and I were brainstorming some ideas about this:
In a typical heavy-write scenario, many store files do not last very long. They're flushed and then within a small number of seconds a compaction runs and they get deleted. For these "short lifetime" store files, it's less likely that a failure will occur during the window in which they're valid. So, I think we can consider some optimizations like the following:
- Flush files at replication count 2. Scan once a minute for any store files in the region that are older than 2 minutes. If they're found, increase their replication to 3. (alternatively, queue them to avoid scanning)
- More dangerous: flush files at replication count 1, but don't count them when figuring log expiration. So, if they get lost, we force log splitting to recover.
The performance gain here is that we avoid the network and disk transfer of writing the third replica for a file that we're just about to delete anyway.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2821) Keep young storefiles at lower
replication
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-2821:
-------------------------------
Attachment: lifetime-distribution.png
Here's a plot of the file lifetimes from an overnight data import. You can see that most of the store files. Here's stats on the distribution:
Min. : 1
1st Qu.: 569
Median : 1353
Mean : 1801
3rd Qu.: 2526
Max. :13342
So, 25% of store files last less than 10 minutes, and half of them last less than 22 minutes.
> Keep young storefiles at lower replication
> ------------------------------------------
>
> Key: HBASE-2821
> URL: https://issues.apache.org/jira/browse/HBASE-2821
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Todd Lipcon
> Attachments: lifetime-distribution.png
>
>
> jgray and I were brainstorming some ideas about this:
> In a typical heavy-write scenario, many store files do not last very long. They're flushed and then within a small number of seconds a compaction runs and they get deleted. For these "short lifetime" store files, it's less likely that a failure will occur during the window in which they're valid. So, I think we can consider some optimizations like the following:
> - Flush files at replication count 2. Scan once a minute for any store files in the region that are older than 2 minutes. If they're found, increase their replication to 3. (alternatively, queue them to avoid scanning)
> - More dangerous: flush files at replication count 1, but don't count them when figuring log expiration. So, if they get lost, we force log splitting to recover.
> The performance gain here is that we avoid the network and disk transfer of writing the third replica for a file that we're just about to delete anyway.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2821) Keep young storefiles at lower
replication
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887998#action_12887998 ]
stack commented on HBASE-2821:
------------------------------
This is a nice idea.
> Keep young storefiles at lower replication
> ------------------------------------------
>
> Key: HBASE-2821
> URL: https://issues.apache.org/jira/browse/HBASE-2821
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Todd Lipcon
> Attachments: lifetime-distribution.png, storefile_age.pl
>
>
> jgray and I were brainstorming some ideas about this:
> In a typical heavy-write scenario, many store files do not last very long. They're flushed and then within a small number of seconds a compaction runs and they get deleted. For these "short lifetime" store files, it's less likely that a failure will occur during the window in which they're valid. So, I think we can consider some optimizations like the following:
> - Flush files at replication count 2. Scan once a minute for any store files in the region that are older than 2 minutes. If they're found, increase their replication to 3. (alternatively, queue them to avoid scanning)
> - More dangerous: flush files at replication count 1, but don't count them when figuring log expiration. So, if they get lost, we force log splitting to recover.
> The performance gain here is that we avoid the network and disk transfer of writing the third replica for a file that we're just about to delete anyway.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2821) Keep young storefiles at lower
replication
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886011#action_12886011 ]
Jonathan Gray commented on HBASE-2821:
--------------------------------------
Sweet graph. Is there something you used to parse the NN log or something to get this from? Would be quite interesting to look at this distribution and the impact of HBASE-2375 and different compaction heuristics being considered.
> Keep young storefiles at lower replication
> ------------------------------------------
>
> Key: HBASE-2821
> URL: https://issues.apache.org/jira/browse/HBASE-2821
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Todd Lipcon
> Attachments: lifetime-distribution.png
>
>
> jgray and I were brainstorming some ideas about this:
> In a typical heavy-write scenario, many store files do not last very long. They're flushed and then within a small number of seconds a compaction runs and they get deleted. For these "short lifetime" store files, it's less likely that a failure will occur during the window in which they're valid. So, I think we can consider some optimizations like the following:
> - Flush files at replication count 2. Scan once a minute for any store files in the region that are older than 2 minutes. If they're found, increase their replication to 3. (alternatively, queue them to avoid scanning)
> - More dangerous: flush files at replication count 1, but don't count them when figuring log expiration. So, if they get lost, we force log splitting to recover.
> The performance gain here is that we avoid the network and disk transfer of writing the third replica for a file that we're just about to delete anyway.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2821) Keep young storefiles at lower
replication
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-2821:
-------------------------------
Attachment: storefile_age.pl
Here's the script I used to parse the NN log.
Then I loaded into R and did something like:
{code}
d <- read.table(file="/tmp/lifetimes-nosplits.txt", header=F)
names(d) <- c("path", "type", "life")
library(ggplot2)
qplot(life, data=subset(d, life<9000), facets=type~., binwidth=10)
{code}
> Keep young storefiles at lower replication
> ------------------------------------------
>
> Key: HBASE-2821
> URL: https://issues.apache.org/jira/browse/HBASE-2821
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: Todd Lipcon
> Attachments: lifetime-distribution.png, storefile_age.pl
>
>
> jgray and I were brainstorming some ideas about this:
> In a typical heavy-write scenario, many store files do not last very long. They're flushed and then within a small number of seconds a compaction runs and they get deleted. For these "short lifetime" store files, it's less likely that a failure will occur during the window in which they're valid. So, I think we can consider some optimizations like the following:
> - Flush files at replication count 2. Scan once a minute for any store files in the region that are older than 2 minutes. If they're found, increase their replication to 3. (alternatively, queue them to avoid scanning)
> - More dangerous: flush files at replication count 1, but don't count them when figuring log expiration. So, if they get lost, we force log splitting to recover.
> The performance gain here is that we avoid the network and disk transfer of writing the third replica for a file that we're just about to delete anyway.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.