[jira] [Commented] (IGNITE-4341) Add TeraSort example as a unit test to Ignite

["Ivan Veselovsky (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722519#comment-15722519 ] 

Ivan Veselovsky commented on IGNITE-4341:
-----------------------------------------

2 problems were solved while incorporating this test:
1) distributed cache mechanism was not working if the Hadoop job submitted with grid().hadoop().submit(); This is related to the fact that real Hadoop client sets job property {code}MRJobConfig.MAPREDUCE_JOB_DIR{code} , while test job submission does not. As a result, org.apache.ignite.internal.processors.hadoop.impl.v2.HadoopV2JobResourceManager#prepareJobEnvironment method did not copy the distributed resources to the local directories, while it should: the "mapreduce.job.dir" property only defines the staging directory, it is not directly related to distributed caching mechanism.
2) Special local file system implementation (org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopLocalFileSystemV1) used in most of Hadoop tests, while original  org.apache.hadoop.fs.LocalFileSystem is used in terasort test. But since the instance of file system is cached in FileSystem.CACHE, an instance from previous tests were actually used. This is fixed by adding org.apache.ignite.internal.processors.hadoop.impl.fs.HadoopFileSystemsUtils#clearFileSystemCache method and invoking it from #beforeTestsStarted and #beforeTest in Hadoop test suites. 

> Add TeraSort example as a unit test to Ignite
> ---------------------------------------------
>
>                 Key: IGNITE-4341
>                 URL: https://issues.apache.org/jira/browse/IGNITE-4341
>             Project: Ignite
>          Issue Type: Test
>          Components: hadoop
>    Affects Versions: 1.7
>            Reporter: Ivan Veselovsky
>            Assignee: Ivan Veselovsky
>             Fix For: 2.0
>
>
> Add canonical TeraSort example as a unit test. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)