[GitHub] [hadoop] elek edited a comment on issue #1065: HDDS-1525. Mapreduce failure when using Hadoop 2.7.5

[GitBox <gi...@apache.org>]

elek edited a comment on issue #1065: HDDS-1525. Mapreduce failure when using Hadoop 2.7.5
URL: https://github.com/apache/hadoop/pull/1065#issuecomment-509627399
 
 
   As it's a bigger patch here is a quick guidance what is included:
   
   ```
    hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java            
    hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzFs.java
   ```
   
   This is the original fix by @xiaoyuyao It makes the RpcClient compatible with older hadoop versions
   
   ```
    hadoop-ozone/dist/src/main/compose/ozone-mr
    hadoop-ozone/dist/src/main/compose/ozone-fs
   ```
   ozone-mr dir is reorganized to have subdirectories for mapreduce tests (hadoop 2.7, 3.1, 3.2).
   Smoketest shell scripts are added
   
   ozone-fs dir is deleted as ozone-mr includes all the functionality and it's better structured.
   
   ```
    hadoop-ozone/dist/src/main/compose/testlib.sh                                                  
   ```
   
   Improved to pass custom robot test arguments. you case see the usage in ozone-mr/hadoop27/test.sh
   
   ```
    hadoop-ozone/dist/src/main/smoketest/ozonefs/hadoopo3fs.robot
   ```
   
   Trival fix, the test was wrong until now.
   
   ```
    hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
    hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
    hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/FileStatusAdapter.java
    hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/FilteredClassLoader.java
    hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java
   ```
   
   This is the second part of the fix (on top of the fixes from @xiaoyuyao. As the FileStatus (hadoop3) is not compatible with FileStatus (hadoop2), we need a class which can be shared between hadoop3 and hadoop2 (FileStatusAdapter, includes only simple java types). 
   
   ```
    hadoop-ozone/ozonefs-lib-legacy/pom.xml
   ```
   
   This is the third part of the fix. The token files should be available for hadoop2 classloader even if they are the cluster is not secure (to serialize/unserialize tokens). The required classes are copied directly to the jar file.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org