You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "Ben Popp (JIRA)" <ji...@apache.org> on 2013/07/10 17:59:48 UTC
[jira] [Commented] (ACCUMULO-884) Take advantage of short circuit
read for local files
[ https://issues.apache.org/jira/browse/ACCUMULO-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704683#comment-13704683 ]
Ben Popp commented on ACCUMULO-884:
-----------------------------------
I was testing Accumulo 1.5 on Hadoop 1.1.2 in order to evaluate the impact of using Fusion IO solid-state drives to accelerate Accumulo random read TP, and I have some anecdotal results.
I added the following properties to my config:
{code}
hdfs-site.xml:
<property>
<name>dfs.block.local-path-access.user</name>
<value>accumulo</value>
</property>
accumulo-site.xml:
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
{code}
Accumulo was able to startup with the new properties, though I did see a lot of the following warnings from various processes
{code}
[conf.ConfigSanityCheck] WARN : BAD CONFIG unrecognized property key (dfs.client.read.shortcircuit)
{code}
In my experiments on these solid state drives, enabling short-circuit reads more than doubled my read throughput! (TP measured in ops/s in a YCSB-derived read-only workload test.)
> Take advantage of short circuit read for local files
> ----------------------------------------------------
>
> Key: ACCUMULO-884
> URL: https://issues.apache.org/jira/browse/ACCUMULO-884
> Project: Accumulo
> Issue Type: Improvement
> Components: docs
> Reporter: Billie Rinaldi
> Assignee: Keith Turner
>
> This is a new feature in hadoop 1.0.x and some versions of 0.22 and 0.23. It allows a client to read directly from disk instead of through a DataNode when the data is stored locally. Enabling it involves setting two configuration parameters, the first in hdfs-site.xml and the second in accumulo-site.xml. We should make sure this works with Accumulo and recommend it in the documentation.
> - dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
> - dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
> See HDFS-2246 and http://hbase.apache.org/book/perf.hdfs.configs.html for more information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira