[jira] [Commented] (HADOOP-11601) Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files

["Daniel Templeton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-11601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284942#comment-15284942 ] 

Daniel Templeton commented on HADOOP-11601:
-------------------------------------------

Thanks, for the patch [~steve_l].

My comments:

{code}
-that is too small will result in either too many jobs being submitted
-for efficient work, or the `JobSubmissionClient` running out of memory.
-
+that is too small will result in badly partitioned workload.
{code}

Your change here correctly states the root effect, whereas what you're removing states the visible side effects.  Seems to me that they're both useful.  Maybe add your statement with the existing statement as a clarifying subordinate phrase?

{code}
        assertPathExists("expected path to be visible before anything written",
            path);
{code}

I don't understand the message here.  Didn't you just write 'a' and then flush it?

{code}
    assertTrue("fs.getDefaultBlockSize(path) size is invalid " + defaultBlockSize,
        defaultBlockSize > 0);
{code}

Would it be helpful here to replace {{path}} with the actual path?

{code}
   * @param parent Test file parent dir path
   *    thrown if an I/O error occurs while writing or reading the test file
   *    <i>other than file not found</i>
   * @return the status
   * @throws IOException
{code}

The throws message does not follow the {{@throws}} tag.

In the {{getFileStatusEventually()}} method, would it be better to include a short (~10ms) sleep to reduce the impact during parallel testing?

> Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-11601
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11601
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, test
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>              Labels: BB2015-05-TBR
>         Attachments: HADOOP-11601-001.patch, HADOOP-11601-002.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> HADOOP-11584 has shown that the contract tests are not validating that {{FileStatus.getBlocksize()}} must be >0 for any analytics jobs to partition workload correctly. 
> Clarify in text and add test to do this. Test MUST be designed to work against eventually consistent filesystems where {{getFileStatus()}} may not be immediately visible, by retrying operation if FS declares it is an object store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org