You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Zhe Zhang (JIRA)" <ji...@apache.org> on 2016/09/10 00:56:20 UTC

[jira] [Created] (HDFS-10854) Remove createStripedFile and addBlockToFile by creating real EC files

Zhe Zhang created HDFS-10854:
--------------------------------

             Summary: Remove createStripedFile and addBlockToFile by creating real EC files
                 Key: HDFS-10854
                 URL: https://issues.apache.org/jira/browse/HDFS-10854
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: erasure-coding, test
    Affects Versions: 3.0.0-alpha2
            Reporter: Zhe Zhang


{{DFSTestUtil#createStripedFile}} and {{addBlockToFile}} were developed before we completed EC client. They were used to test the {{NameNode}} EC logic when the client was unable to really create/read/write EC files.

They are causing confusions in other issues about {{NameNode}}. For example, in one of the patches under {{HDFS-10301}}, {{testProcessOverReplicatedAndMissingStripedBlock}} fails because in the test we fake a block report from a DN, with a randomly generated storage ID. The DN itself is never aware of that storage. This is not possible in a real production environment.
{code}
      DatanodeStorage storage = new DatanodeStorage(UUID.randomUUID().toString());
      StorageReceivedDeletedBlocks[] reports = DFSTestUtil
          .makeReportForReceivedBlock(block,
              ReceivedDeletedBlockInfo.BlockStatus.RECEIVING_BLOCK, storage);
      for (StorageReceivedDeletedBlocks report : reports) {
        ns.processIncrementalBlockReport(dn.getDatanodeId(), report);
      }
{code}

Now that we have a fully functional EC client, we should remove the old testing logic and use similar logic as non-EC tests (creating real files and emulate blocks missing / being corrupt).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org