You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jay Booth (JIRA)" <ji...@apache.org> on 2009/07/31 22:54:14 UTC
[jira] Created: (HDFS-516) Low Latency distributed reads
Low Latency distributed reads
-----------------------------
Key: HDFS-516
URL: https://issues.apache.org/jira/browse/HDFS-516
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Jay Booth
Priority: Minor
I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: hdfs-516-20090831.patch
New patch, IPC server was too slow for IO operations (like 40 times slower than DFS without caching) so I wrote a custom ByteServer that's streamlined to avoid object creation or byte copying whenever possible and defaults to tcp nodelay. Client connections pool using commons-pool. Uses static methods in hdfs.rad.ByteServiceProtocol for all serialization, faster than reflection. On the laptop in pseudodistributed, I'm seeing 5X faster than DFS for random searches.
Refactored a bunch on the client side, eliminated a few redundant classes, still need to make lookahead happen via a separate thread in caching byteservice and tweak a couple things in ByteServer for performance, then this thing will be pretty fast. I'm gonna run some numbers on EC2 tonight/tomorrow and see what I come up with.
Also, cleaned up unit tests to JUnit 4 and added some javadoc, probably missed a bunch of places and could certainly expand on all of it. Haven't added license to the header of every file yet, license explicitly granted here, will get to that for next patch.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090824.patch, hdfs-516-20090831.patch, radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: (was: hdfs-516-20090831.patch)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: (was: radfs.patch)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: hdfs-516-20090912.patch
New patch:
* README file with instructions for eclipse and running with hadoop
* Javadoc and JUnit 4-style test cases, some new test cases
* Benchmarks for random reads, binary search, and streaming
* Illustrated 100% performance increase in streaming case, somehow, from 213 seconds to 112 seconds to stream 1GB from a remote HDFS file. Reproduced a couple times, using 16MB of cache with the lookahead mechanism. I suspect it uses a lot more CPU than conventional streaming, but still, that's a lot faster.
* No longer requires any change to HDFS code, module is now entirely in contrib
* Cleans up file handles better
* Handles remote disconnect better from the client side
What are people's thoughts on getting this into 0.21? It shows a lot of promise as far as performance but hasn't been tested on larger clusters, I'd be confident up to 200 nodes or so and then I'd start getting nervous.
Given that it lives entirely in contrib and needs to be actively configured to turn it on, could we include this for 0.21? Anyone want to try running the benchmarks?
I'll run the benchmarks one last time tomorrow to sanity check the latest patch (changed a couple things since the last time I ran in a cluster), then maybe we could consider committing?
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: radfs.patch
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: radfs.patch
Full patch file from hdfs-trunk/.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch, radfs.patch, radfs.tgz
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755121#action_12755121 ]
Todd Lipcon commented on HDFS-516:
----------------------------------
Jay: that sounds good to me. Let's explicitly mark the API as "Experimental" and "Likely to disappear in a future release with little or no warning - use at your own risk" :) I think especially as security and authentication begin to take form in the next couple months it will be a headache to try to maintain this code.
If branching in SVN weren't such a pain, I'd suggest we maintain this in a separate branch that didn't go into releases... c'est la vie ;-)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738445#action_12738445 ]
Todd Lipcon commented on HDFS-516:
----------------------------------
Any benchmarks for this? Would be interesting to see what the improvement actually is for various read patterns.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756264#action_12756264 ]
Todd Lipcon commented on HDFS-516:
----------------------------------
bq. Unless you want to add checksums for better comparison, I don't think it is every essential.
I disagree here - checksums are responsible for a reasonable amount of the overhead in the current HDFS implementation. If this is mostly seen as a testing ground for performance improvements, we can't be comparing apples to oranges. If we don't want to implement checksums in RadFs, then the other option for fair comparison is to remove checksums from Hdfs.
If this is about a testing ground for new features, then that makes sense, but I understood this mostly as a "turbo HDFS client experimentation"
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: (was: radfs.tgz)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: hdfs-516-20090824.patch
Changed caching system to ehcache, which gives us a configurable api with better caching options. Fixed all known correctness bugs and added lots of test coverage.
Got SequenceFileSearcher working, but it depends on HADOOP-6196 -- I could fold the searcher code and searcher test into HADOOP-6196 if people are interested in the functionality.
Added benchmarking utils, .sh scripts intended to be run from HADOOP_HOME/bin.
Note, if anyone installs, run ant bin-package from project root before trying to run radfs unit tests.
TODO:
benchmark
javadoc, pretty up codebase, change unit tests to JUnit 4 style, add filesystem contract unit test
I'll get a fork going on github soon if anyone's interested in trying it out.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090824.patch, radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Status: Patch Available (was: Open)
Here's the initial patch. I have some pretty decent byte-consistency and integration tests wrapping it but no actual applications built using it. I started on a SequenceFileSearcher that implemented binary search but dealing with the sync points was sticky so it's not working yet (and commented out, with associated failing test). I'm hoping to get that working in the next couple weeks when I have some time as an example along with doing a performance comparison between nutch FsDirectory using DistributedFileSystem and RadFileSystem to see if this shows the gains that I'm thinking it will. My only change to core HDFS was to add getFile() to FSDatasetInterface so that the RadNode (plugged into datanode via ServicePlugin) can open FileChannels.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738680#action_12738680 ]
Raghu Angadi commented on HDFS-516:
-----------------------------------
Yes, client cache surely helps with accesses like binary search. Client cache certainly has a place.
It is fine and very useful for you continue your work without worrying too much about the integration for now.
You might need add basic CRC support for better comparison with default DFS in benchmarks, but probably not a must for first cut. Note that current RPC has LOT of buffer copying overhead (both on client and server). Since the I/O benchmarks are not CPU bound it would not show up but it is an important factor for production loads.
> [...] So while I agree with the urge for simplicity, I feel like we need to make that performance tradeoff clear. Otherwise, we could have a lot of very slow mapreduce jobs happening. [...]
Of course, we would not let that happen.. I think, in longer term, we will have streaming access almost as good as now and random access with improved latency.. fairly transparent to the user.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737799#action_12737799 ]
Jay Booth commented on HDFS-516:
--------------------------------
Ok, thanks, is one big patch preferred to the tiny patch + tarball?
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch, radfs.tgz
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738651#action_12738651 ]
Jay Booth commented on HDFS-516:
--------------------------------
Wow, thanks Raghu, that's awesome and will save me a ton of time. A couple points for discussion:
* The random 4k byte grabber is awesome and I will be using it as part of my benchmarking at the first opportunity, however I think it's worth also testing some likely applications to really show the strength of client-side caching. 10MB or so worth of properly warmed cache could mean your first 20 lookups in a binary search are almost-free, and having the frontmost 10% of a lucene index in cache will mean that almost all of the scoring portion of the search will be computed against local memory. Meanwhile, for truly random reads, having a cache that's, say, 5-10% of the size of the data will only get you a small improvement. So I'd like to get some numbers for use cases that really thrive on caching in addition to truly random access. But that will be extremely useful for tuning the IO layer and establishing a baseline for cache-miss performance, so thanks for the heads up.
* I have a feeling that my implementation is significantly slower than the default when it comes to streaming, since it relies on successive, small positioned reads and a heavy memory footprint rather than a simple stream of bytes. Watching my unit tests run on my laptop with a ton of confounding factors sure seemed that way, although that's not a scientific measurement (one more item to benchmark). So while I agree with the urge for simplicity, I feel like we need to make that performance tradeoff clear. Otherwise, we could have a lot of very slow mapreduce jobs happening. Given that MapReduce is the primary use case for Hadoop, my instinct was to make RadFileSystem a non-default implementation. Point very well taken about the BlockLocations and CRC verification, maybe the best way to handle future integration with DataNode would be to develop separately, reuse as much code as possible and then when RadFileSystem is mature and benchmarked we can revisit a merge with DistributedFileSystem?
Thanks again, I'll try and write a post later tonight with an explicit plan for benchmarking and then maybe people can comment and poke holes in it as they see fit?
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755640#action_12755640 ]
Todd Lipcon commented on HDFS-516:
----------------------------------
bq. I essentially see this as POC of what could/should be improved in HDFS for addressing latency issues. Contrib makes sense, but I would not expect this to go to production in this form and should be marked 'Experimental'.
+1
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755627#action_12755627 ]
Raghu Angadi commented on HDFS-516:
-----------------------------------
bq. somehow, from 213 seconds to 112 seconds to stream 1GB from a remote HDFS file.
This is 5MBps for HDFS and 9MBps for RadFS. Assuming 9MBps is probably 100Mbps network limit (is it?), 5MBps is too low for any FS. Since both reads are from the same physical files, this may not be hardware related. Could you check what is causing this delay? This might be affecting other benchmarks as well. Checking netstat on the client while this read is going on might help.
Regd reads in RAD fs, does client fetch 32KB each time (single RPC) or does it pipeline (multiple requests for single client's stream)?
@Todd, I essentially see this as POC of what could/should be improved in HDFS for addressing latency issues. Contrib makes sense, but I would not expect this to go to production in this form and should be marked 'Experimental'. The benchmarks also help greatly in setting priorities for features. I don't think this needs a branch since it does not touch core at all.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755823#action_12755823 ]
Jay Booth commented on HDFS-516:
--------------------------------
Yeah, I was puzzled by the performance too. I dug through the DFS code and I'm saving a bit on new socket and object creation, maybe a couple instructions here and there, but that shouldn't add up to 100 seconds for a gigabyte (approx 20 blocks). I'm calling read() a bajillion times in a row so it's conceivable (although unlikely) that I'm pegging the CPU and that's the limiting factor.
I'm busy for a couple days but will get back to you with some figures from netstat, top and whatever else I can think of, along with another streaming case that works with read(b, off, len) to see if that changes things. I'll do a little more digging into DFS as well to see if I can isolate the cause. I definitely did run them several times on the same machine and another time on a different cluster with similar results, so it wasn't simply bad luck on the rack placement on EC2 (well maybe but unlikely).
Will report back when I have more numbers. After I get those, my roadmap for this is to add checksum support and better DatanodeInfo caching. User groups would come after that.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: radfs.odp
Here's my presentation at HadoopWorld tomorrow if anyone's interested
Short version:
still faster than HDFS for random reads
my erstwhile-fast streaming was entirely because I wasn't checksumming - checksumming slows things down a lot
my checksumming implementation is ghetto - simply wrapped ChecksummingFileSystem around the whole thing, so it's slow
includes plan to implement PipeliningByteService as another configurable part of the current ByteService chain -- would prefetch pages and checksum them in a separate thread
I'll get a new patch up soon as well as a github repo, don't have the code with my presently. It looks like with implementation of PipeliningByteService I'll be able to equal/surpass HDFS for streaming while keeping the lead in random reads and generating a lower server-side workload.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch, radfs.odp
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: radfs.tgz
Oops. Shows what I get for posting at the end of the day, svn diff misses new files, duh. Here's a tarball of src/contrib/radfs. build.xml should play well with the rest of hadoop
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch, radfs.tgz
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738565#action_12738565 ]
Raghu Angadi commented on HDFS-516:
-----------------------------------
Jay, random read is an (increasingly more) important feature for HDFS to support. Currently latency is the biggest draw back. See HDFS-236. It is good to see your work on this. You could also run simple benchmark in HDFS-236 that does simple random read on a file and does not depend on a sequence file.
>From your architecture description this reduces the latency through following improvements :
* Connection caching (Through RPC).
* File Channel caching on Server
* Local cache on the client.
These are complementary to existing datanode. I might be a lot more simpler to add these features to existing implementation rather than requiring a user to choose an implementation based on the access. As such you will have to re-implement many features (BlockLocations on client, CRC verification, effcient bulk transfers AVRO-24, etc )
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738462#action_12738462 ]
Konstantin Boudnik commented on HDFS-516:
-----------------------------------------
Perhaps the common requirements aren't applicable to the contrib work, but I would like to add few review comments:
- The content of TestSequenceFileSearcher.java is commented out. Should this file be included in the patch at all?
- There's a push to upgrade existing test base to Junit4 (see HADOOP-4901 for more information). I see that the tests are developed in old JUnit3 fashion (TestCase extension, lack of @Test notation, junit.framework.* packages, etc.) Although JUnit4 will pickup these tests without any problems, I'd suggest that new tests are compliant with JUnit4.
- a number of classes have a lack of JavaDocs even for public classes/methods
- the content of SequenceFileSearcher.java is commented out. Is it need to be a part of this patch at all?
- pieces of the code here and there (e.g. ConcurrentLRUCache.java) are commented out. Should they be just removed from the patch?
- some classes don't have standard Apache license boiler plate (ByteService.java, ByteServiceLazyInitializer.java, and many other)
- FSDatasetInterface is being changed: shall this information be added to the release notes?
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
This work didn't really go anywhere, I tried multithreading on the client side to make a request-per-packet approach viable, but it wound up being even slower than singlethreaded request-per-packet. If there's interest in the client-side byte cache, I could create another issue with just that improvement.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch, radfs.odp
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737806#action_12737806 ]
Tsz Wo (Nicholas), SZE commented on HDFS-516:
---------------------------------------------
> Ok, thanks, is one big patch preferred to the tiny patch + tarball?
Yes. Otherwise, the automated build won't work.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch, radfs.tgz
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737788#action_12737788 ]
dhruba borthakur commented on HDFS-516:
---------------------------------------
This sounds interesting. The patch you posted includes only the changes needed to Hadoop code, it does not have your implementation of the RadFileSystem.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: radfs.patch
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738488#action_12738488 ]
Jay Booth commented on HDFS-516:
--------------------------------
* The content of TestSequenceFileSearcher.java is commented out. Should this file be included in the patch at all?
** Yeah, I need to get that working and said so in the initial comment -- I think that once it's working, it would make sense to include it in the patch as a useful piece of functionality that doubles as an example use-case, but I'll certainly defer to the community's sensibilities on that.
* There's a push to upgrade existing test base to Junit4 (see HADOOP-4901 for more information). I see that the tests are developed in old JUnit3 fashion (TestCase extension, lack of @Test notation, junit.framework.* packages, etc.) Although JUnit4 will pickup these tests without any problems, I'd suggest that new tests are compliant with JUnit4.
** Will do. I actually deliberately made them 3-like because that looked like the existing convention, but if we're trying to change that, I completely agree.
* a number of classes have a lack of JavaDocs even for public classes/methods
** Doh, yep, need to fix that for sure. Just wanted to post the JIRA for feedback before I was unable to work on it this week, apologies.
* the content of SequenceFileSearcher.java is commented out. Is it need to be a part of this patch at all?
** Same as above on its testcase -- it doesn't need to be part of the patch but once I get it working, it would be a useful tool
* pieces of the code here and there (e.g. ConcurrentLRUCache.java) are commented out. Should they be just removed from the patch?
** Yeah, any commented out code in the patch is the result of my oversight, will absolutely fix at my first opportunity. ConcurrentLRUCache was stolen from Solr so it's those guys' fault ;) I'll clean out the commented code though and while I'm at it, I'll port over their unit test for it.
* some classes don't have standard Apache license boiler plate (ByteService.java, ByteServiceLazyInitializer.java, and many other)
** Will add
* FSDatasetInterface is being changed: shall this information be added to the release notes?
** Yeah, it's a pretty minor change that I noted in my initial post, should I make a README.txt in contrib/radfs? Alternatively, I could work with FSDataset directly instead of FSDatasetInterface (FSDataset exposes getFile(), the interface doesn't), which would mean zero lines of changed code in HDFS core at the expense of some tighter coupling. Willing to go either way at the discretion of the community.
Thanks a ton for the feedback, if anyone else has anything at all I'd love to hear it before I get back to working on this later in the week.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755033#action_12755033 ]
Raghu Angadi commented on HDFS-516:
-----------------------------------
Hi Jay,
will go through the patch. I hope a few others get a chance to look at it as well.
Since it is contrib, it certainly makes it easier to include in trunk. I am not sure about 0.21 timeline.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: (was: radfs.patch)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756214#action_12756214 ]
Raghu Angadi commented on HDFS-516:
-----------------------------------
When you get a change please point me to the streaming test/benchmark.
bq. After I get those, my roadmap for this is to add checksum support and better DatanodeInfo caching. User groups would come after that.
Unless you want to add checksums for better comparison, I don't think it is every essential. You need not spend much time on getting feature parity with HDFS.
For more users to benefit from your work, I think it is better to extract the features that are complementary to HDFS. and we can work on getting those into HDFS.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: (was: radfs.patch)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750246#action_12750246 ]
Jay Booth commented on HDFS-516:
--------------------------------
I did some benchmarking, here are the results:
Each test ran 1000 searches to warm, then 5000 searches to benchmark.
Binary search of a 20GB sorted sequence file of 20 million 1kb records.
Tests were run from the namenode in a 4-node EC2 medium cluster, 1.7 GB of ram each. 1 namenode and 3 datanodes.
>From HDFS to a 512MB cached RadFS there was a 4X average improvement in search times, from 102ms to 24ms.
Each search was, theoretically, 24.25 reads (log 2 of 20 million). Not actually measured.
I only ran each set once. The 90th percent line trends the right way, although the max line is a little spikey. I'll add a 99th % in future benchmarks.
HDFS, baseline:
Warming with 1000 searches
Executed 5000 random searches with FS class org.apache.hadoop.hdfs.DistributedFileSystem
Done, Search Times:
Mean: 102.17840000000015
Variance: 5939.660105461091
Median: 97.0
Max: 3095.0
Min: 33.0
90th pct: 130.0
Rad, no cache
Executed 5000 random searches with FS class org.apache.hadoop.hdfs.rad.RadFileSystem
Done, Search Times:
Mean: 68.55640000000002
Variance: 233.8335857571515
Median: 67.0
Max: 379.0
Min: 26.0
90th pct: 79.0
Rad, 16MB cache:
Warming with 1000 searches
Executed 5000 random searches with FS class org.apache.hadoop.hdfs.rad.RadFileSystem
Done, Search Times:
Mean: 42.039799999999985
Variance: 237.83818359671966
Median: 40.0
Max: 203.0
Min: 5.0
90th pct: 59.0
Rad, 128MB cache:
Warming with 1000 searches
Executed 5000 random searches with FS class org.apache.hadoop.hdfs.rad.RadFileSystem
Done, Search Times:
Mean: 29.850600000000007
Variance: 202.08189601920367
Median: 27.0
Max: 203.0
Min: 1.0
90th pct: 45.0
Rad, 512MB cache:
Warming with 1000 searches
Executed 5000 random searches with FS class org.apache.hadoop.hdfs.rad.RadFileSystem
Done, Search Times:
Mean: 24.274600000000014
Variance: 250.3052558911758
Median: 22.0
Max: 687.0
Min: 0.0
90th pct: 36.0
I could still shave a point or two by cleaning up my caching system to be more graceful with its lookahead mechanism, but not bad for now. I'll pretty it up and post a first attempt at a final patch soon.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090824.patch, hdfs-516-20090831.patch, radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737713#action_12737713 ]
Jay Booth commented on HDFS-516:
--------------------------------
Here's some architectural overview and a general request for comments on the matter, I'll be away and busy the next few days but should be able to get back to this in the middle of next week.
The basic workflow is I created a RadFileSystem (RandomAccessDistributed FS) which wraps DistributedFileSystem and delegates to it for everything except for getFSDataInputStream. That returns a custom FSDataInputStream which wraps a CachingByteService which itself wraps a RadFSByteService. The caching byte services share a cache which is managed by the RadFSClient class (could maybe factor that away and put it in RadFileSystem instead). They try to hit the cache, and if they miss, they call the underlying RadFSClientByteService to get the requested page plus a few pages of lookahead. The RadFSClientByteService calls the namenode to get appropriate block locations (todo, cache these effectively) and then calls RadNode, which is embedded in DataNode via ServicePlugin and maintains an IPCServer and a set of FileChannels to the local blocks. On repeated requests for the same data, the RadFSClient tends to favor going to the same host, figuring that the benefit of hitting the DataNode's OS cache for the given bytes outweighs the penalty of hopping a rack in terms of reducing latency (untested assumption).
The intended use case is pretty different from MapReduce so I think this should be a contrib module that has to be explicitly invoked by clients. It really underperforms DFS in terms of streaming but should (haven't tested extensively outside of localhost) significantly outperform it in terms of random reads. In terms of files with 'hot paths', such as lucene indices or binary search over a normal file, cache hit percentage is likely to be pretty high so it should probably perform pretty well. Currently, it makes a fresh request to the NameNode for every read, which is inefficient but more likely to be correct. Going forward, I'd like to tighten this up, make sure it plays nice with append and get it into a future Hadoop release.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738455#action_12738455 ]
Jay Booth commented on HDFS-516:
--------------------------------
I have some obligations this week but will hopefully get back to this over the weekend and put together some benchmarks. I'm thinking I'll benchmark binary search over 20GB and 100GB sequence files (could be a semi-replacement for mapfile in certain circumstances?) and lucene search using nutch's FsDirectory implementation. I should have something up by the 10th.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757192#action_12757192 ]
Jay Booth commented on HDFS-516:
--------------------------------
Hey Raghu, the streaming benchmark is in org.apache.hdfs.rad.benchmark.Streaming -- there's a .sh file there that's used to invoke it, if you follow the directions in the README and then copy those .sh files over to your HADOOP_COMMON_HOME/bin directory, they should work.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755120#action_12755120 ]
Jay Booth commented on HDFS-516:
--------------------------------
Hey Todd, in short, I agree, we should be looking at moving performance improvements over to the main FS implementation. Right now, my version doesn't support user permissions or checksumming. I'd say it makes sense to keep it in contrib as a sandbox for now, and work towards full compatibility with the main DFS implementation at which point we could consider swapping in the new reading subsystem? User permissioning would require some model changes but should be workable, checksumming probably won't be too bad if I read the code right.
So, I suppose keep it in contrib as a sandbox initially with an explicit goal of moving it over to DFS when it reaches compatibility? It doesn't really lend itself to moving over piecemeal, as it has several components which all pretty much need each other. However, it's pretty well integrated with the DFS API and only replaces one method on the filesystem class.
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: (was: hdfs-516-20090824.patch)
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737797#action_12737797 ]
Tsz Wo (Nicholas), SZE commented on HDFS-516:
---------------------------------------------
> .., svn diff misses new files, ...
For new files, run "svn add /path/to/new/files" before "svn diff".
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: radfs.patch, radfs.tgz
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755048#action_12755048 ]
Todd Lipcon commented on HDFS-516:
----------------------------------
I haven't had a chance to look over the patch as of yet, but I have one concern:
Is there a plan for deprecation in the event that HDFS itself achieves similar performance? I think having an entirely separate FS implementation that only differs in performance is not a good idea longterm. Using this contrib project as an experimentation ground sounds great, but I think long term we should focus on improving DistributedFileSystem's performance itself, and not bifurcate the code into the "fast version that we dont really support because it's contrib" and "slow version that we do".
I'll try to find a chance to look over the patch soon, but in the meantime do you have any thoughts on the above?
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Attachments: hdfs-516-20090912.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-516) Low Latency distributed reads
Posted by "Jay Booth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Booth updated HDFS-516:
---------------------------
Attachment: radfs.patch
Attached wrong patch before, fixed
> Low Latency distributed reads
> -----------------------------
>
> Key: HDFS-516
> URL: https://issues.apache.org/jira/browse/HDFS-516
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Jay Booth
> Priority: Minor
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I created a method for low latency random reads using NIO on the server side and simulated OS paging with LRU caching and lookahead on the client side. Some applications could include lucene searching (term->doc and doc->offset mappings are likely to be in local cache, thus much faster than nutch's current FsDirectory impl and binary search through record files (bytes at 1/2, 1/4, 1/8 marks are likely to be cached)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.