You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Eric Newton (JIRA)" <ji...@apache.org> on 2012/05/07 16:48:50 UTC
[jira] [Created] (ACCUMULO-578) consider using hdfs for the walog
Eric Newton created ACCUMULO-578:
------------------------------------
Summary: consider using hdfs for the walog
Key: ACCUMULO-578
URL: https://issues.apache.org/jira/browse/ACCUMULO-578
Project: Accumulo
Issue Type: Improvement
Components: logger, tserver
Affects Versions: 1.5.0-SNAPSHOT
Reporter: Eric Newton
Assignee: Eric Newton
Using HDFS for walogs would fix:
* ACCUMULO-84: any node can read the replicated files
* ACCUMULO-558: wouldn't need to monitor loggers
* ACCUMULO-544: log references wouldn't include hostnames
* ACCUMULO-423: wouldn't need to monitor loggers
* ACCUMULO-258: hdfs has load balancing already
To implement it, we would need the ability to distribute log sorts.
Continuing to use loggers helps us avoid:
* hdfs pipeline strategy
* we don't have fine-grained insight when a single node makes dfs slow
* additional namenode pressure
* flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Adam Fuchs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Fuchs updated ACCUMULO-578:
--------------------------------
Attachment: HDFS_WAL_states.pdf
State diagram for new HDFS WAL.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Keith Turner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280514#comment-13280514 ]
Keith Turner commented on ACCUMULO-578:
---------------------------------------
It seems like these changes will work. Given the following events.
# TserverA creates Walog1
# User deletes lock for TserverA
# GC does not see TServerA in zookeeper
# GC does not see any references to Walog1 in !METADATA
# TserverA writes that TabletX is using Walog1 to !METADATA
# TserverA writes mutations for TabletX to Walog1
# TserverA sends confirmation of write to client
# TserverA notices its lock went away and kills itself
It must be possible to interleave the following two GC events in any order after event 4 above and not have data loss.
# GC recovers lease on Walog1
# GC looks for Walog1 in !METADATA
If recovering the lease will make writes by other processes fail after that point in time, I think its ok. With this assumption all interleavings should result in an error writing to the walog or the GC seeing the walog in !METADATA. An error writing the walog would prevent the confirmation from being sent to the client. If writes can occur after lease recover, then there is an interleaving that will result in data loss.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Attachment: comparison.png
After many attempts, I've attached a good picture of the performance difference.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (ACCUMULO-578) consider using hdfs for the
walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280091#comment-13280091 ]
Eric Newton edited comment on ACCUMULO-578 at 5/21/12 1:58 PM:
---------------------------------------------------------------
The loggers implemented another feature I forgot about which was to atomically determine which logs were not open, and therefore available for collection. We need a new way to determine this.
# open the file and begin using it, in a directory named for the tserver id
# write references to the log into the !METADATA table, as the log is used
# tablet server removes references to logs as tablets flush to disk
# tserver deletes the log when the last reference is removed
# gc doesn't collect files with good tserver ids or METADATA table references
# master will assign log sorts when it finds an unassigned tablet with log references
# once a tablet's logs have been sorted, the tablet is assigned by the master
# gc will remove sorted log entries when all references to the logs have been removed
For backwards compatibility, tservers will copy/sort any existing logger-based WALs to the recovery area when they start.
was (Author: ecn):
The loggers implemented another feature I forgot about which was to atomically determine which logs were not open, and therefore available for collection. We need a new way to determine this.
h3. One approach:
# write references to the log into the !METADATA table, along with a tserver id
# open the file and begin using it
# when the tablet server closes the log, it removes the tserver id
# gc doesn't collect files with good tserver ids or METADATA table references
h3. Bonus points:
# move the majority of log file collection to the tablet servers: they know when their tablets have lost references to the log
# master/tservers can sort the logs during recovery, and delete the unsorted copy
# tablet servers can gc the recovery log, after a check of the METADATA table
#* Need to avoid problems like ACCUMULO-598
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Attachment: (was: comparison.png)
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Attachment: comparison.png
Fixed the key
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (ACCUMULO-578) consider using hdfs for the
walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280276#comment-13280276 ]
Eric Newton edited comment on ACCUMULO-578 at 5/21/12 5:48 PM:
---------------------------------------------------------------
I wasn't aware of it. I should make the prototype use a -threadpool- different thread for writing and run another test.
was (Author: ecn):
I wasn't aware of it. I should make the prototype use a threadpool for writing and run another test.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280091#comment-13280091 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
The loggers implemented another feature I forgot about which was to atomically determine which logs were not open, and therefore available for collection. We need a new way to determine this.
h3. One approach:
# write references to the log into the !METADATA table, along with a tserver id
# open the file and begin using it
# when the tablet server closes the log, it removes the tserver id
# gc doesn't collect files with good tserver ids or METADATA table references
h3. Bonus points:
# move the majority of log file collection to the tablet servers: they know when their tablets have lost references to the log
# master/tservers can sort the logs during recovery, and delete the unsorted copy
# tablet servers can gc the recovery log, after a check of the METADATA table
#* Need to avoid problems like ACCUMULO-598
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280393#comment-13280393 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
I think the tserver write to the log will fail because the recovery code calls the name node to recover the lease.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (ACCUMULO-578) consider using hdfs for the
walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on ACCUMULO-578 started by Eric Newton.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) use hdfs for the walog
Posted by "Keith Turner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-578:
----------------------------------
Affects Version/s: (was: 1.5.0-SNAPSHOT)
Fix Version/s: 1.5.0
> use hdfs for the walog
> ----------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Reporter: Eric Newton
> Assignee: Eric Newton
> Fix For: 1.5.0
>
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Attachment: comparison.png
12-hour continuous ingest test runs on a 10-node test cluster, with the # of entries plotted over time. The one ingesting faster is using HDFS for the WAL.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Attachment: NNOpsComparison.pdf
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Attachment: (was: comparison.png)
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278795#comment-13278795 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
Preliminary tests of a prototype shows that writing to HDFS is 10-15% faster than writing to specialized loggers. This is better than I expected.
I'll post the results of a 12-hour continuous ingest run when it completes.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282602#comment-13282602 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
More thinking about file GC: we can eliminate the possibility of removing a log file that is in use by asking the tserver to remove the log, instead of doing it in the GC.
# open the file and begin using it, in a directory named for the tserver address
# write references to the log into the !METADATA table, as the log is used
# tablet server removes references to logs as tablets flush to disk
# gc asks the tserver to remove the file when it sees no METADATA table references
#* the tablet server ignores the request if it is still using the log
# master will assign log sorts when it finds an unassigned tablet with log references
#* log sorts need to recover the lease on the file to prevent stray updates from appearing
#* log sorts should be monitored, perhaps made into a FATE operation
# once a tablet's logs have been sorted, the tablet is assigned by the master
# gc will remove sorted log entries when all references to the logs have been removed
# as always, checks against the !METADATA table have to use the special consistency checking iterators
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269809#comment-13269809 ]
Todd Lipcon commented on ACCUMULO-578:
--------------------------------------
bq. we don't have fine-grained insight when a single node makes dfs slow
See HDFS-3170 for some ideas here.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-578) use hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-578:
---------------------------------
Summary: use hdfs for the walog (was: consider using hdfs for the walog)
> use hdfs for the walog
> ----------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280439#comment-13280439 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
After some thought, it sounds like we need to do this:
# open the file and begin using it, in a directory named for the tserver id
# write references to the log into the !METADATA table, as the log is used
# tablet server removes references to logs as tablets flush to disk
# tserver deletes the log when the last reference is removed
# gc doesn't collect files with good tserver ids or METADATA table references
# master will assign log sorts when it finds an unassigned tablet with log references
#* log sorts need to recover the lease on the file to prevent stray updates from appearing
# once a tablet's logs have been sorted, the tablet is assigned by the master
# gc will remove sorted log entries when all references to the logs have been removed
#* gc needs to recover the lease on the file, and then double-check the metadata table for references
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (ACCUMULO-578) use hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton resolved ACCUMULO-578.
----------------------------------
Resolution: Fixed
> use hdfs for the walog
> ----------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281195#comment-13281195 ]
Todd Lipcon commented on ACCUMULO-578:
--------------------------------------
Great! That about matches up with the improvement we saw in HBase when we implemented this feature. We're now experimenting with multiple parallel WALs per server to stripe across more disks: getting another 50% or so from that in some tests.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280276#comment-13280276 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
I wasn't aware of it. I should make the prototype use a threadpool for writing and run another test.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280939#comment-13280939 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
I did a quick measurement of the number of NN operations done over a short ingest period, to evaluate if logging to HDFS would significantly increase pressure on the NN.
I pre-split the ingest table into 8 tablets and used goraci to ingest 80 million entries (240M k-v). I grabbed the jmx counters on the NN before the start of the map-reduce job, and then 10 minutes later. That was enough time for the idle flush and the GC to run a couple of times.
As expected, the change was negligible.
I don't have a good sense of what the different costs of the different Ops might be... but nothing stands out as horribly different.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Eric Newton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281103#comment-13281103 ]
Eric Newton commented on ACCUMULO-578:
--------------------------------------
Todd, preliminary testing shows a considerable improvement interleaving writes with flushes.
|Logger rate|~432K|
|HDFS rate|~487K|
|HDFS with parallel flush|~699K|
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, NNOpsComparison.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280272#comment-13280272 ]
Todd Lipcon commented on ACCUMULO-578:
--------------------------------------
Nice stuff! You guys are aware of HDFS-895, right? ie you can pipeline multiple calls to hflush() to eliminate "bubbles" in the write path.
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-578) consider using hdfs for the walog
Posted by "Keith Turner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280369#comment-13280369 ]
Keith Turner commented on ACCUMULO-578:
---------------------------------------
Eric,
I was looking at the proposed GC algorithm and trying to think of situations where the following might occur.
# In use walog is deleted
# Unused walog is never deleted
It looks pretty solid. The following interleaving of events could be problematic. This is possible because there is time between when a lock is deleted and when a tablet server kills itself.
# TserverA creates Walog1
# User deletes lock for TserverA
# GC does not see TServerA in zookeeper
# GC does not see any references to Walog1 in !METADATA
# TserverA writes that TabletX is using Walog1
# TserverA notices its lock went away and kills itself
# GC deletes Walog1
# TabletX fails to load because Walog1 does not exists
> consider using hdfs for the walog
> ---------------------------------
>
> Key: ACCUMULO-578
> URL: https://issues.apache.org/jira/browse/ACCUMULO-578
> Project: Accumulo
> Issue Type: Improvement
> Components: logger, tserver
> Affects Versions: 1.5.0-SNAPSHOT
> Reporter: Eric Newton
> Assignee: Eric Newton
> Attachments: HDFS_WAL_states.pdf, comparison.png
>
>
> Using HDFS for walogs would fix:
> * ACCUMULO-84: any node can read the replicated files
> * ACCUMULO-558: wouldn't need to monitor loggers
> * ACCUMULO-544: log references wouldn't include hostnames
> * ACCUMULO-423: wouldn't need to monitor loggers
> * ACCUMULO-258: hdfs has load balancing already
> To implement it, we would need the ability to distribute log sorts.
> Continuing to use loggers helps us avoid:
> * hdfs pipeline strategy
> * we don't have fine-grained insight when a single node makes dfs slow
> * additional namenode pressure
> * flexibility: for example, we can add fadvise() calls to the logger before HDFS supports it
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira