[jira] [Issue Comment Deleted] (HDFS-4849) Idempotent create and append operations.

["Konstantin Shvachko (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-4849:
--------------------------------------

    Comment: was deleted

(was: > As regards creating blocks simultaneously, it is prevented precisely because multiple creates are not allowed today and hence the application does not even get to allocating blocks.

My example is different. You create file once. THEN spawn 10 threads to add blocks in the same file.
To demonstrate my point I wrote a snippet below. Just past&copy it into your favourite test and run. You will be amazed what you will get, I was. Also try to vary the {{bufLen}}. Some interesting things happen when you get to the second block.
The system can be abused and this has nothing to do with the retries.
{code}
  static volatile int nrDone;
  public static void main(String[] args) throws Exception {
    Configuration conf = new HdfsConfiguration();
    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
        numDataNodes(3).build();
    FileSystem fs = cluster.getFileSystem();
    Path path = new Path("testFile.txt");
    final FSDataOutputStream out = fs.create(path);
    nrDone = 0;
    for(int i = 0; i < 10; i++) {
      new Thread() {
        @Override
        public void run() {
          int bufLen = 128*1024*1024 + 7;
          byte[] toWrite = new byte[bufLen];
          try {
            out.write(toWrite, 0, toWrite.length);
          } catch (IOException e) {
            LOG.error("Failed to write a string: ", e);
          }
          nrDone++;
        }
      }.start();
    }
    while(nrDone < 10) ;
    out.close();
    LOG.info("Done. " + fs.getFileStatus(path));
    for(BlockLocation bl : fs.getFileBlockLocations(path, 0, Long.MAX_VALUE))
      LOG.info(bl);
  }
{code})
    
> Idempotent create and append operations.
> ----------------------------------------
>
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>         Attachments: idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce chances for a job or other app failures when NN fails over.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira