[jira] [Commented] (BIGTOP-867) Add integration tests that verify filesystem support for pig, hive, etc

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598997#comment-13598997 ] 

Steve Loughran commented on BIGTOP-867:
---------------------------------------

>From the HADOOP-8545 work, we've started writing integration tests that use OpenStack swift as a source and destination for Pig, MR and ultimately, Hive work. These verify that the filesystem are compatible at the MR layer.
https://github.com/hortonworks/Hadoop-and-Swift-integration/tree/hwx/swiftfs-integration
These tests cannot go into hadoop-common with HADOOP-8545, because they depend on many downstream artifacts. If they were made more generic they could be used to test many external filesystems, including Amazon S3 and FTP -which cannot be tested by setting fs.default.name and running existing test suites, because they are not conventional filesystems.
                
> Add integration tests that verify filesystem support for pig, hive, etc
> -----------------------------------------------------------------------
>
>                 Key: BIGTOP-867
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-867
>             Project: Bigtop
>          Issue Type: New Feature
>          Components: Tests
>    Affects Versions: 0.6.0
>            Reporter: Steve Loughran
>
> Add tests that can use non-HDFS filesystems as the source and destination of MR jobs, so test the blobstore client libraries that ship with Hadoop. These tests could also be run against HDFS, or by developers of other filesystems

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira