[jira] [Commented] (HDFS-8254) In StripedDataStreamer, it is hard to tolerate datanode failure in the leading streamer

["Tsz Wo Nicholas Sze (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-8254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589204#comment-14589204 ] 

Tsz Wo Nicholas Sze commented on HDFS-8254:
-------------------------------------------

Thanks Zhe for the review.

>   ... Can we add some Javadocs for the class and methods, and ideally, a design description on the JIRA?

Sure, will add the following javadoc.
{code}
  /**
   * This class is designed for multiple threads to share a
   * {@link MultipleBlockingQueue}. Initially, the queue is empty. The earliest
   * thread calling poll populates entries to the queue and the other threads
   * will wait for it. Once the entries are populated, all the threads can poll
   * their entries.
   *
   * @param <T> the queue entry type.
   */
{code}

> ... It also looks like we could run into a race condition ...

As Walter explained, there is no race condition.

> DFSStripedOutputStream already has the schema ...

The JIRA is for replacing leading streamer implementation.  For the other issues, let's fix them in separated JIRAs.

> New methods in DataStreamer could use some Javadoc.

Do you mean callUpdateBlockForPipeline and callUpdatePipeline?  These internal APIs are both short and straightforward.  I believe javadoc is not needed.

>  class Coordinator could be renamed to something like StreamersCoordinator, just to be more specific.

The class actually is DFSStripedOutputStream.Coordinator.

> The Javadoc of StripedBlockUtil#checkBlocks should say that it checks the two blocks are in the same block group

Will fix it.

> In StripedDataStreamer, it is hard to tolerate datanode failure in the leading streamer
> ---------------------------------------------------------------------------------------
>
>                 Key: HDFS-8254
>                 URL: https://issues.apache.org/jira/browse/HDFS-8254
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h8254_20150526.patch, h8254_20150526b.patch
>
>
> StripedDataStreamer javadoc is shown below.
> {code}
>  * The StripedDataStreamer class is used by {@link DFSStripedOutputStream}.
>  * There are two kinds of StripedDataStreamer, leading streamer and ordinary
>  * stream. Leading streamer requests a block group from NameNode, unwraps
>  * it to located blocks and transfers each located block to its corresponding
>  * ordinary streamer via a blocking queue.
> {code}
> Leading streamer is the streamer with index 0.  When the datanode of the leading streamer fails, the other steamers cannot continue since no one will request a block group from NameNode anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)