[jira] Updated: (HADOOP-1942) Increase the concurrency of transaction logging to edits log

["dhruba borthakur (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1942:
-------------------------------------

    Attachment: transactionLogSync3.patch

This patch includes the locking changes to optimize writing and sync-ing of the edit log. It also includes statistics to gather the following:

1.Number of transactions
2.Time to write these transactions to memory buffer (average& total)
3. Number of syncs
4. Time to do these syncs (average & total)

These statistics are written to the Namenode log once every minute. They are also written to the statistics aggregator daemon if present.

This patch includes a unit-test that creates 100 threads and each thread processes 1000 transactions. For this test case, the current trunk does about 95000 syncs. Trunk plus this patch does about 4000 syncs. A huge improvement!

> Increase the concurrency of transaction logging to edits log
> ------------------------------------------------------------
>
>                 Key: HADOOP-1942
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1942
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.15.0
>
>         Attachments: transactionLogSync.patch, transactionLogSync2.patch, transactionLogSync3.patch
>
>
> For some typical workloads, the throughput of the namenode is bottlenecked by the rate of transactions that are being logged into tghe edits log. In the current code, a batching scheme implies that all transactions do not have to incur a sync of the edits log to disk. However, the existing batch-ing scheme can be improved.
> One option is to keep two buffers associated with edits file. Threads write to the primary buffer while holding the FSNamesystem lock. Then the thread release the FSNamesystem lock, acquires a new lock called the syncLock, swaps buffers, and flushes the old buffer to the persistent store. Since the buffers are swapped, new transactions continue to get logged into the new buffer. (Of course, the new transactions cannot complete before this new buffer is sync-ed).
> This approach does a better job of batching syncs to disk, thus improving performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.