[jira] [Commented] (HDFS-7782) Read a striping layout file from client side

["Zhe Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341408#comment-14341408 ] 

Zhe Zhang commented on HDFS-7782:
---------------------------------

I had an offline discussion with [~cmccabe] today around several options to support striped reading:
# We can simply create multiple {{DFSInputStream}} and let each read fixed-indexed cells in each stripe. For example, under a 6+3 schema and 1MB striping cell size, the first {{DFSInputStream}} is in charge of reading ranges 0~1MB, 6~7MB, etc. Then we have a special wrapper input stream to coordinate them.
# We can also create another striped {{BlockReader}}. This direction has been briefly explored under HDFS-7653. 
# We can also extend {{DFSInputStream}} itself to work with multiple regular {{BlockReader}} instances.

Option #1 is the least intrusive to existing code but also the least efficient. {{DFSInputStream}} is a rather heavyweight class with many internal states to store and maintain; it is an overkill to use it to read a given range from a given location. For example, each {{DFSInputStream}} queries located blocks from NameNode.

The choice between #2 and #3 is about maximizing code reuse. We are leaning toward #3 because the _failover_ and _hedged read_ logics in current {{DFSInputStream}} are closely tied to _contiguous replicated_ format and need to be updated even in option #2.

Below is a plan:
# Refactor {{DFSInputStream}} (preferably in trunk)
#* Separate out the client-DN block reading logic into a subclass {{DFSContiguousInputStream}}
#* Keep in {{DFSInputStream}} most internal states (except {{blockReader}}), client-NN communications, as well as wrapping logics such as {{readWithStrategy}}
# Create a {{DFSStripedInputStream}} subclass with:
#* An array of {{blockReaders}}
#* The logic of parsing a {{LocatedStripedBlock}} and assigning target address and block ID to each {{blockReader}}
#* Decoding logic for failover

[~jingzhao] and [~szetszwo] I don't think we covered this issue in detail in the meetups. Could you take a look and share some advice? Thanks!

> Read a striping layout file from client side
> --------------------------------------------
>
>                 Key: HDFS-7782
>                 URL: https://issues.apache.org/jira/browse/HDFS-7782
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Li Bo
>            Assignee: Zhe Zhang
>
> If client wants to read a file, he is not necessary to know and handle what layout the file is. This sub task adds logic to DFSInputStream to support reading striping layout files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)